Razib Khan One-stop-shopping for all of my content

September 4, 2017

The GRE is useful; range restriction is a thing

Filed under: Intelligence,Psychometrics — Razib Khan @ 8:05 pm

The above figure is from Beyond the Threshold Hypothesis: Even Among the Gifted and Top Math/Science Graduate Students, Cognitive Abilities, Vocational Interests, and Lifestyle Preferences Matter for Career Choice, Performance, and Persistence. It shows that even at very high levels of attainment on standardized tests there are differences in life outcome based on variation. The old joke is that results on intelligence tests don’t matter beyond a certain point…that point being whatever your own position is! But these results show that mathematics SAT outcomes at age 13 can still predict a lot of things across a wide range.

From personal experience people outside of psychology are pretty unaware of the power of cognitive aptitude testing. This includes many biologists. I was reminded of the above figure as I read portions of Richard Haier’s The Neuroscience of Intelligence. If you are a biologist curious about the topic, this is a highly recommended book.

The main reason I am posting this is that this it was asked of me by a friend in academia. There has recently been a backlash against the GRE exam, with support from the highest echelons of the science media. Additionally, many researchers in public forums are voicing objections to the GRE very vocally. Naturally this has resulted in counterarguments…but respondents have to be very careful how the couch their disagreement, because they fear being accused of being racist, sex, or classist. Such accusations might trigger social media mobs, which no one wants to be the target of (and if past experience is any guide, friends and colleagues will stand aside while the witch is virtually burned, hoping to avoid notice).

Because of the request above I finally decided to look at the two papers which are eliciting the current wave of GRE-skepticism, The Limitations of the GRE in Predicting Success in Biomedical Graduate School and Predictors of Student Productivity in Biomedical Graduate School Applications. To my eye they suffer from the same problem as all earlier criticisms: range restriction.

The issue is that if a university is using the GRE and other metrics well as filters for those admitted then there shouldn’t be that much variation in outcome left (the outcome being publications or some other important metric which actually leads to the production of science, as opposed to test scores and grades). The two papers above look at those admitted to biomedical programs at UNC and Vanderbilt, while another study looked at UCSF. These are all universities with standards high enough that there are either explicit or implicit cut-off scores so that many students are removed from the applicant pool immediately (the mean scores are well above the 50th percentile, you can see them in the paper yourself).

When I was in graduate school I was on a fellowship committee for several years, and I had access to GRE scores and grades. But I didn’t really pay much attention to them because there wasn’t that much range. And to be honest if the student was beyond their first year I didn’t look at all as time went on. In contrast, I did look really closely at the recommendations from their advisors. From talking to others on the committee this seemed typical. Once students were admitted they were judged based on how they were doing in graduate school. And how they were doing in graduate school had to do with research, not their graduate school GPA or what they scored on the GRE to get in.

As an empirical matter I do think that it is likely many universities will follow the University of Michigan in dropping the GRE as a requirement. There will be some resistance within academia, but there is a lot of reluctance to vocally defend the GRE in public, especially from younger faculty who fear the social and professional repercussions (every time a discussion pops up about the GRE I get a lot of Twitter DMs). My prediction is that after the GRE is gone people will simply rely on other proxies.

If the GRE is not required, but can be taken, then students who do well on the GRE will put that on their application. Sometimes strong students encounter tragedies in their undergraduate years which strongly impact their grade point averages, and very strong GREs can help show admissions committees that they can do the coursework despite their undergraduate record (I’m not positing a hypothetical, but recounting real individuals I’ve known of and seen). It seems cruel to deny these students the chance to submit their test scores. This means that those professors who believe the GRE is valid will show preference to students who take the test and have strong scores (and to be sure, many more care about the GRE when it means someone concretely joining their lab, as opposed to the abstraction of who gets admitted to the department).

More broadly, professors who are taking students will look more at proxies for GRE score, such as undergraduate institution, or the prestige of the recommendation letters. In some places, such as Britain, standardized testing emerged in part as a way to identify strong students from underprivileged backgrounds. These are not the type of students who would ever be able to present a prestigious letter of recommendation. This is a sort of student which still exists (often they are from non-academic backgrounds, being the first to graduate from college in their family; what they lack in polish they compensate for in aptitude, but that take the right environment to express).

The recourse to other variables besides the GRE score will likely have mixed results at best. Consider the successful campaign to ban asking for job applicants’ criminal records. It turns out that just increased discrimination against all young black men, because employers could not longer differentiate. In general I think removing the GRE would probably hurt graduates of less prestigious state universities the most if I had to guess (and of course students from East Asia, who tend to have a comparative advantage on standardized tests). I’m pretty sure we’ll see, as the experiment will be run.

Addendum: There are professors at relatively prestigious research universities who had mediocre or sub-par GRE scores. We all know them. To some extent I think many of these individuals almost take pride in the fact that they accomplished so much in science despite negative feedback due to their impressive test scores. But remember that we’re talking about trends and averages, not deterministic predictions.

April 13, 2017

The coming reign of the Baby Boomer gerontocracy

Filed under: Gerontocracy,Psychology,Psychometrics — Razib Khan @ 1:22 pm

From Dawn to Decadence: 1500 to the Present: 500 Years of Western Cultural Life is one of my favorite books. It’s one of those works whose breadth and depth is such that I would recommend it to anyone. Jacques Barzun began writing this work when he was 84, and it was published in his 93rd year. Born in 1907 Barzun saw the full efflorescence of 20th century Western culture across much of its span firsthand. When people say that when you age you gain wisdom, surely in the domain of scholarship Barzun’s production in the last few decades of his life would qualify.

But not everyone is Jacques Barzun. If you read Intelligence: All That Matters or peruse some of Eliott Tucker-Drob’s work you will know that cognitive function declines with age beyond your twenties. Different subcomponents may decline at different rates. And, they decline differently in different people (e.g., some people may develop dementia, so their faculties will decline far faster at an earlier age). But, by and large any gains in experience or wisdom are going to be balanced against declines in raw analytic ability, as well as the slow entropic loss of information.

This is not an inconsequential matter. Our governing class is quite old. The average age in Congress may be 55 to 60, but it is almost certainly true that more senior members with more power and authority are older. The president of the United States is 70 years old. If you look at the plots in these figures by 70 there has been a notable drop in intelligence by this age, though again, it may vary from person to person.

But most important in light of these figures is that the Supreme Court is a lifetime appointment, and many of its members are quite old, an anticipate serving until they are quite old if they are younger. In the mid-1970s justice William O. Douglas had a stroke and was basically not mentally competent to serve. Because of this fact, and Douglas’ reluctance to retire his fellow justices basically did not take his vote into account. Three of the justices today are over the age of 70, with Clarence Thomas nearing that age, and two are over the age of 80.

When it comes to Congress, or even the President, there seems to be some sort of institutional support as well as the larger collective vote in the case of Congress, which might buffer the cognitive impact of a gerontocracy. But aside from law clerks Supreme Court justices have to rely on their own individual mental capacities.

The Mormon Church has a gerontocracy among its we openleadership. Even my most devout friends in the church sometimes found it amusing how old their leadership was, and how quickly they died in succession due to the seniority principle. But The Supreme Court is not the leadership of a relatively small church. It impacts our whole nation. This sort of gerontocracy is no laughing matter.

Will we openly speak of the age issue? I doubt it. Today the Baby Boomers are between the ages of 53 an 71. They are coming into their own as a cohort into the highest reaches of the gerontocracy. If there is any generation with the grace and humility to step aside for the greater good, it will not be this generation.

August 17, 2011

Looking for a few good 145+ I.Q. individuals

Above is the distribution of self-reported I.Q.s of the readers of this weblog according to the 2011 survey. I point this out because my friend Steve Hsu will be giving a talk at Google later today. Here are the details:

I’ll be giving a talk at Google tomorrow (Thursday August 18) at 5 pm. The slides are here. The video will probably be available on Google’s TechTalk channel on YouTube.

The Cognitive Genomics Lab at BGI is using this talk to kick off the drive for US participants in our intelligence GWAS. More information at www.cog-genomics.org, including automatic qualifying standards for the study, which are set just above +3 SD. Participants will receive free genotyping and help with interpreting the results. (The functional part of the site should be live after August 18.)

Title: Genetics and Intelligence

Abstract: How do genes affect cognitive ability? I begin with a brief review of psychometric measurements of intelligence, introducing the idea of a “general factor” or IQ score. The main results concern the stability, validity (predictive power), and heritability of adult IQ. Next, I discuss ...

May 13, 2011

I.Q. and genomics

Filed under: B.G.I.,Behavior Genetics,Genomics,I.Q.,Psychology,Psychometrics — Razib Khan @ 9:15 pm

In my experience most scientists are not too clear on the details of intelligence testing, perhaps because the whole area is somewhat in ill repute (except when you want to brag about your own SAT/GRE score!). This despite the fact that the profession of science is skewed toward the right end of the intelligence bell curve. Steve Hsu, a physicist at the University of Oregon (and someone I’ve known for a while in the interests of “full disclosure”) has a nice presentation up in PDF format which summarizes the major points of interest in this area. Worth a skim if you are unfamiliar. Additionally he alludes to future directions in the study of the genetic basis of intelligence using genomics. Here’s his abstract:

I begin with a brief review of psychometric results concerning intelligence (sometimes referred to as the g factor, or IQ). The main results concern the stability, validity (predictive power) and heritability of adult IQ. Next, I discuss ongoing Genome Wide Association Studies which investigate the genetic basis of intelligence. Due mainly to the rapidly decreasing cost of sequencing (currently below $5k per genome), it is likely that within the next 5-10 years we will identify genes ...

September 27, 2010

American family values: where even the dull can dream!

ResearchBlogging.orgOne of the issues when talking about the effect of environment and genes on behavioral and social outcomes is that the entanglements are so complicated. People of various political and ideological commitments tend to see the complications as problems for the other side, and yet are often supremely confident of the likely efficacy of their predictions based on models which they shouldn’t even been too sure of. That is why cross-cultural studies are essential. Just as psychology has overly relied on the WEIRD nature of data sets, so it is that those interested in social science need to see if their models are robust across cultures (I’m looking at you economists!).

That is why this ScienceDaily headline, Family, Culture Affect Whether Intelligence Leads to Education, grabbed my attention. The original paper is Family Background Buys an Education in Minnesota but Not in Sweden:

Educational attainment, the highest degree or level of schooling obtained, is associated with important life outcomes, at both the individual level and the group level. Because of this, and because education is expensive, the allocation of education across society is an important social issue. A dynamic quantitative environmental-genetic model can help document the effects of social allocation patterns. We used this model to compare the moderating effect of general intelligence on the environmental and genetic factors that influence educational attainment in Sweden and the U.S. state of Minnesota. Patterns of genetic influence on educational outcomes were similar in these two regions, but patterns of shared environmental influence differed markedly. In Sweden, shared environmental influence on educational attainment was particularly important for people of high intelligence, whereas in Minnesota, shared environmental influences on educational attainment were particularly important for people of low intelligence. This difference may be the result of differing access to education: state-supported access (on the basis of ability) to a uniform higher-education system in Sweden versus family-supported access to a more diverse higher-education system in the United States.


Minnesota is to some extent the Scandinavia of America, so the cross-cultural difference is particularly notable. You wouldn’t be surprised for example by big differences between Mississippi and Sweden. But looking at a comparison between the Upper Midwest and Scandinavia is closer to seeing the impact of national culture and policy differences on populations which were originally very similar.

Their methodology was simple, though as with much of this sort of behavior genetic work the statistical analysis can be somewhat labored. In both Sweden and Minnesota you had samples of dizygotic and monozygotic twins which give you a way to compare the effect of genes on variation in life outcomes. Sweden has large data sets from male conscription for behavior genetics analysis. They compared this with the Minnesota Twin Family Study data set.

Since the topline results are pretty straightforward, I thought I’d give you some statistics. Table 1 has raw correlations. Note that they converted educational attainment into a seven-point scale, less than 9 years of education to completion of doctoral studies.

swed1

You see the expected drop off in correlation between identical and fraternal twins. Identical twins share more genetic identity than fraternal twins, so they’re going to be more identical on a host of metrics aside from appearance. Those are just raw correlation values of traits though across categories of twins. The core intent of the paper was to explore the relationship between genes, family environment, and other environmental factors, and educational attainment. To do this they constructed a model. Below you see estimates of the variance in the trait explained by variance in genes, shared environment (family), non-shared environment (basically “noise” or error, or it could be something like peer group), from Sweden to Minnesota, and, at three intelligence levels. Two standard deviations below the norm is borderline retarded, ~2.5% of the population or so, and two standard deviations above the norm is at Mensa level.

swed2

It’s interesting that as you move up the IQ scale the genetic variation explains more and more of variance the educational attainment. Someone with an IQ of 130 is likely to be university educated. But there are many who are not. Why? The way I interpret these results is that if you are that intelligent and do not manage to complete university you may have heritable dispositions of personality which result in you not completing university. If, for example, you come from a family which is very intelligent, but is low on conscientiousness, then there may be a high likelihood that you just won’t complete university because you can’t be bothered to focus. Or, you may have personality characteristics so that you don’t want to complete university. A second major finding here is that Sweden and the USA go in totally different directions when it comes to the sub-average and dull in prediction of years of education. Why? The explanation in the paper seems plausible: Sweden strongly constrains higher education supply, but makes it available to those with proven academic attainments at a nominal price. Family encouragement and connections don’t matter as much, if you can’t pass the university entrance examination you can’t pass it. In contrast in the USA if you’re dull, but come from a more educated or wealthier family, you can find some university or institution of higher education which you can matriculate in. Supply is more flexible to meet the demand. I actually know of a woman who is strongly suspected to be retarded by her friends. I have been told she actually tested in the retarded range in elementary school but was taken out of that  track because her family demanded it (she’s the product of a later conception, and her family made their money in real estate, not through professional advancement). Over the years she has enrolled in various community colleges, but never to my knowledge has she completed a degree. If she had not had family connections there is a high probability she wouldn’t have completed high school. As it is, she can check off “some college” on demographic surveys despite likely be functionally retarded.

The next table is a bit more confusing. It shows you the correlations between the effects of the variable on education and intelligence. In other words, does a change in X have the same directional effect on Y and Z, and what is the extent of the correspondence between the effect on Y and Z.

swed3

Shared environment had almost the same effect on intelligence and education, while genetics had a more modest simultaneous effect. Not too surprising that non-shared environment didn’t have a strong correlation in effect across the traits, the authors note that much of this is going to noise in the model, and so not systematically biased in any direction. Though the confidence intervals here are a little strange. I’m not going to get into the details of the model, because frankly I’m not going to replicate the analysis with their data myself. That’s why I wanted to present raw correlations first. That’s pretty straightforward. Estimates of variances out of models with a set of parameters is less so. Here’s an interesting point from the correlations in the last table:

The patterns of genetic correlations in the two samples differed. In Sweden, genetic correlation was steadily in excess of .50 across the range of intelligence, indicating a genetically influenced direct effect of intelligence on educational attainment that was weaker than the shared environmental effect on educational attainment. In the MTFS [Minnesota] population, however, genetic correlation was in excess of .50 when level of intelligence was low, but was halved at higher levels of intelligence. This indicated that genetic influences on intelligence tended to limit educational attainment when the level of intelligence was low, but not when the level of intelligence was average or high.

Now let me jump to conclusion:

This finding indicates that genetic influences common to intelligence and educational attainment may have been more effective in limiting educational attainment in individuals with low levels of intelligence than in encouraging educational attainment in those with high levels of intelligence. As in Sweden, shared environmental influences on intelligence and educational attainment were completely linked, indicating a direct contribution from shared environmental influences on intelligence to educational attainment. The decrease in shared environmental variance with higher intelligence, however, indicated that shared environmental influences were more effective in encouraging educational attainment in higher-intelligence individuals than in limiting educational attainment in lower-intelligence individuals. In other words, in populations in which shared environmental influences such as family history and values encouraged high levels of educational attainment, individuals were able to surmount limitations in intelligence.

Our analysis does not permit the conclusion that these differences in educational systems cause the differences in environmental and genetic influences on educational attainment observed in this study, but it is reasonable to hypothesize that this is likely. In particular, the greater expense of higher education and greater subjectivity of admission standards in the United States compared with Sweden may partially explain the very different patterns of shared environmental influences in the two population samples. Regardless of the causes underlying the differences we observed, the results of our study make clear that the degrees of environmental and genetic influences can vary substantially between groups with different circumstances, and even within such groups. Our results also suggest that the ways in which social systems are organized may have implications for how and to what extent environmental and genetic influences on behavior will be expressed.

This discussion about the role of environment, genes, and culture, on various outcomes should not hinge on one paper. But, these sorts of results are often not widely disseminated among the intellectual classes. One aspect of the American educational system in contrast to some other nations is that not-too-brights have university degrees. Education has long been a project for social engineering in the USA, going back to Horace Mann. Legacies, underrepresented minorities, the poor, those with particular talents, etc., are all part of the decentralized system of university admissions in the United States. In contrast, in nations such as Sweden or Japan there is a more centralized and universal set of criteria. This results is more perfect sorting by the metrics of interest without considerations of social engineering. I know that Sweden has traditionally had a small aristocratic class, while the Japanese aristocracy were basically abolished after World War II. Additionally, both are relatively homogeneous societies so considerations of racial representativeness are not operative. Or weren’t until recently in the case of Sweden. But consider one reality: if such a system is perfectly meritocratic over time if the traits being evaluated are heritable then you will have genetic stratification and reduction of social mobility assuming assortative mating at university.

Currently there is some handwringing by the elites about the fact that so few poor kids get admitted to Ivy League universities. I think there’s a simple way to change this: get rid of the implicit Asian quotas. After all, there was a lot of socioeconomic diversity after the Ivy League universities got rid of their Jewish quotas, but the children of the Jews who didn’t have to go to CUNY and went to Harvard are well off themselves. But more socioeconomic mobility through removing the implicit Asian quota would cause other difficulties, as elite private universities need their slots for both legacies as well as underrepresented minorities for purposes of social engineering/fostering diversity/encouraging donations. Additionally, just as with the Jews the welter of mobility in one generation of the children of Asian immigrants would settle into quiescence in the next if the traits which enable university admission are genetically or culturally heritable.

Citation: Johnson W, Deary IJ, Silventoinen K, Tynelius P, & Rasmussen F (2010). Family background buys an education in Minnesota but not in Sweden. Psychological science : a journal of the American Psychological Society / APS, 21 (9), 1266-73 PMID: 20679521

June 27, 2010

Psychometrics, epigenetics and economics

Filed under: IQ,Psychology,Psychometrics — Razib Khan @ 10:26 am

Two papers of interest. IQ in the Production Function: Evidence from Immigrant Earnings (ungated). And Human Intelligence and Polymorphisms in the DNA Methyltransferase Genes Involved in Epigenetic Marking. My impression is that the focus on epigenetics has a higher-order social motive; even the sort of humanists who are involved with N + 1 have asked me about the topic. But how many people know what methylation is?

Powered by WordPress