Drawing inferences from data is limited by what the data measure
October 15, 2010 Leave a comment
In “Why Genomics Falls Short as a Medical Tool,” Matt Ridley points out how tracking genetic associations hasn’t yielded as much explanatory power as hoped to inform medical applications:
It’s a curious fact that genomics has always been sold as a medical story, yet it keeps underdelivering useful medical knowledge and overdelivering other stuff. … True, for many rare inherited diseases, genomics is making a big difference. But not for most of the common ailments we all get. Nor has it explained the diversity of the human condition in things like height, intelligence and extraversion.
He notes that even something as straightforward and heritable as height has been difficult to predict from the genes identified:
Your height, for example, is determined something like 90% by the tallness of your parents—so long as you and they were decently well fed as children. … In the case of height, more than 50 genetic variants were identified, but together they could account for only 5% of the heritability. Where was the other 95%?
Some may argue that it’s a case of needing to search more thoroughly for all the relevant genes:
A recent study of height has managed to push the explained heritability up to about half, by using a much bigger sample. But still only half.
Or, perhaps there are so many genetic pathways that affect height that it would be difficult to identify and generalize from them all:
Others… think that heritability is hiding in rare genetic variants, not common ones—in “private mutations,” genetic peculiarities that are shared by just a few people each. Under this theory, as Tolstoy might have put it, every tall person would be tall in a different way.
Ridley closes by emphasizing that genes influence outcomes through complex interactions and network effects.
If we expect education research and application to emulate medical research and application, then we need to recognize and beware of its limitations as well. Educational outcomes are even more multiply determined than height, personality, and intelligence. If we seek to understand and control subtle environmental influences, we need to do much more than simply measure achievement on standardized tests and manipulate teacher incentives.