Razib Khan One-stop-shopping for all of my content

November 9, 2017

Patterns in international GRE scores

Filed under: GRE,Psychometrics — Razib Khan @ 12:52 am

Why writing up my earlier post I stumbled onto to some interesting GRE data for applicants for various countries. I transcribed the results for all nations with sample sizes greater than 500. What you see above is a plot which shows mean quantitative and verbal scores on the GRE by nations.

The correlation in this set of countries between subtests of the GRE are as so:

Quant & verbal = 0.33

Verbal & writing = 0.84

Quant & writing = 0.21

Basically, the writing score and verbal score seem to reflect the lack of English fluency in many nations.

Many of these results are not too surprising if you’ve ever seen graduate school applications in the sciences (I have). Applicants from the United States tend to have lower quantitative and higher verbal scores. This is what you see here. It’s rather unfair since the test is administered in English, and that’s the native language of the United States. No surprise the United Kingdom and Canada score high on verbal reasoning. Ireland, Australia, and New Zealand didn’t have enough test takers to make the cut, but they all do as well as the United Kingdom. Singapore has an elite group which uses English as the medium of instruction in school.

I didn’t include standard deviation information, even though it’s in there. India has a pretty high standard deviation on quantitative reasoning, at 9.1. In contrast, China only has a standard deviation of 5.2 for quantitative reasoning. More than twice as many Indians as Chinese take the GRE.

Finally, I want to observe Saudi Arabia, as opposed to Iran. Both countries have about 5,000 people taking the GRE every year. About 2.5 times as many people live in Iran as opposed to Saudi Arabia. But the results for Saudi Arabia are dismal, while Iranian students perform rather well on the quantitative portion of the GRE.* This is not surprising to me, having seen applications from Saudi and Iranian students.

Saudi Arabia wants to move beyond being purely a resource-driven economy. These sorts of results show why many people are skeptical: in the generations since the oil-boom began the Saudi state has not cultivated and matured the human capital of its population. To get a better sense, here are the scores with N’s of MENA nations and a few others:

Country N Quantitative
Saudi 4462 141.6
Libya 113 146.2
Iraq 148 146.6
Oman 98 146.9
UAE 238 147.2
Qatar 85 147.3
Kuwait 386 147.8
Algeria 86 149.5
Yemen 68 149.9
Bahrain 55 150.9
Ethiopia 353 151.3
Jordan 472 152.1
Egypt 1044 153.2
Morocco 191 153.7
Tunisia 128 154.1
Georgia 71 154.2
Lebanon 691 154.7
Armenia 84 154.9
Azerbaijan 125 155.1
Eritrea 223 155.2
Israel 344 156.8
Iran 5319 157.3
Turkey 2370 158.9

 The “natural break” is between the Saudis and everyone else. In recent years Saudis indigenized their non-essential workforce. I’m broadly skeptical of the consequences of this.

The data for the plot at the top is below the fold.

Country Verbal Quant Writing
China 148.4 165.6 3
Taiwan 147.1 162.2 2.9
Hong Kong 150.2 161.1 3.4
Singapore 157.8 160.4 4.3
S Korea 149.9 160.3 3.2
Vietnam 147.6 159.1 3.2
Turkey 144.9 158.9 2.9
Japan 146.4 158.2 3.1
Iran 143.5 157.3 2.8
France 154 157.1 3.5
Greece 150.4 156.7 3.6
Germany 153.7 156.3 3.8
Thailand 144.7 156.2 2.9
Russia 149.3 156.1 3.2
Malaysia 150.8 155.8 3.6
Bangladesh 144.8 155.7 2.9
Italy 154.8 155.6 3.4
Sri Lanka 144.4 155.4 3.1
Chile 150.5 155.3 3.1
Nepal 144.8 155 3
Spain 152.3 155 3.4
Lebanon 147.5 154.7 3.3
Canada 156.1 154.6 4.2
UK 157.4 154.1 4.3
Egypt 145 153.2 3.1
India 144 153.2 2.9
Pakistan 147.9 152.5 3.4
Indonesia 147 152.2 3.1
Brazil 150.3 152.2 3.1
Philippines 150.7 150.9 3.6
Ecuador 147.4 150.5 3.1
USA 152.8 150.2 3.8
Colombia 148.6 150.1 3
Mexico 148.9 149.5 3.1
Kenya 147.3 147.8 3.4
Ghana 146.1 147.4 3.2
Nigeria 146.4 146.9 3.1
Saudi Arabia 137.5 141.6 2

* I suspect the poor English language skills of Iranian students is partly a function of the nation’s isolation the past two generations, but that’s speculation on my part.

November 8, 2017

GRE utility for graduate school and conditioning on the dependent variable

Filed under: GRE,Psychometrics — Razib Khan @ 8:43 pm

One of the things that seems to be popular in biological sciences right now is the push to get rid of the GRE as part of the criteria for entrance. Two of the major rationales are that it’s expensive, so discriminates against lower socioeconomic status candidates, and, that it makes it harder to recruit underrepresented minorities since on average they score lower on the GRE (many departments have either explicit or implicit GRE cut-offs).

I’m not going to litigate these issues. To be honest I believe it is a fait accompli that many departments will stop using the GRE. This will probably increase diversity in some ways. But I also suspect it will result in a greater bias toward more “polished” candidates since very high GRE scores sometimes indicate to admissions committees that applicants who are otherwise spotty or irregular may have promise.

But, I do want to enter into the record a major problem with the argument that GRE does not correlate with academic success at the graduate level (supported by research). Yes, part of the issue may simply be range restriction. But there is another issue which many biological scientists may not be familiar with.

First, right now this paper from early this year is getting a lot of attention, The Limitations of the GRE in Predicting Success in Biomedical Graduate School.

It was, of course, a political scientist who objected immediately:

This blog post is of interest for those curious, That one weird third variable problem nobody ever mentions: Conditioning on a collider. Basically, it is well known that at many universities graduate admittees exhibit a weak negative association between GRE scores and grade point averages. This was commented on as far back as the 1970s in ScienceGraduate Admission Variables and Future Success:

The standard variables considered in selecting students for graduate school do not correlate well with later measures of the success or attainments of the selected students (1, 2). The low correlations have led at least one investigator (3) to propose abandoning one of these standard variables, the Graduate Record Examination (GRE). The purpose of the present report is to demonstrate that variables that are the basis for admitting students to graduate school must have low correlations with future measures of the success of these students.

What’s going on?

As noted in the paper there are some universities which are first-choices for graduate school in a field to such an extent that they will admit candidates who have very high GPAs and very high GREs. In this case, neither of the criteria will predict success because there is very little variation to generate a correlation. But, at many universities, there is a negative correlation between admittee GRE score and undergraduate GPA. That is because very few applicants will be admitted with both low GRE and GPA scores, but some will be admitted with high GRE scores and low(er) GPAs and others with higher GPAs and low(er) GREs (usually there is still a GPA and GRE floor).

Consider the relation:

    \[ R^2 = \frac{r_1^2 + r_2^2 - 2r_1r_2r}{1 - r^2} \]

Where \R^2 is the proportion of the variance of the variable you want to predict, and r_1^2 and r_2^2 are the correlations between GRE and GPA and that the variable of interest, and r is the correlation between GRE and GPA.

Basically, when you have negative correlations you’re going to get into a situation where r_1^2 and r_2^2 are not going to be able to explain a lot of the variance in what you want to predict.

This may seem like a nerdy issue. And it is well known to social scientists. But since the people I see talking about the GRE are academics in the biological sciences I thought I would at least highlight this nerdy issue.

As I said above, I do think GRE is going to be dropped as a requirement at many universities for graduate programs. This is going to be a natural experiment, so we’ll be able to test many hypotheses. The paper above ends like so:

…Without a study in which a sample of the applicants-rather than of the selected students is evaluated, it is impossible to tell [the validity of the criteria -RK]. Yet such a study is completely infeasible. Even if rejected applicants are monitored throughout the rest of their working careers, it is impossible to evaluate how they would have done had they been admitted, because the rejection itself constitutes an important “treatment” difference between them and the selected students. The alternative is to admit a sample of the applicant population without using the standard admission variables to select them-preferably, to select at random.

Selection may not be random, but I believe we may be able to test some hypotheses in the next generation by testing a set of students later on after admittance on the GRE and see what the future correlation is.

January 25, 2012

Classicists are smart!

Filed under: Data Analysis,GRE,Intelligence,Social Science — Razib Khan @ 4:15 am

The post below on teachers elicited some strange responses. Its ultimate aim was to show that teachers are not as dull as the average education major may imply to you. Instead many people were highly offended at the idea that physical education teachers may not be the sharpest tools in the shed due to their weak standardized test scores. On average. It turns out that the idea of average, and the reality of variation, is so novel that unless you elaborate in exquisite detail all the common sense qualifications, people feel the need to emphasize exceptions to the rule. For example, over at Fark:

Apparently what had happened was this: He played college football. He majored in math, minored in education. When he went to go get a job, he took it as a math teacher. When the football coach retired/quit, he took over. When funding for an advance computer class was offered, he said he could teach it after he got the certs – he easily got them within a month.

So the anecdote here is a math teacher who also coached. Obviously the primary issue happens to be physical education teachers who become math teachers! (it happened to me, and it happened to other readers apparently) In the course of double checking the previous post I found some more interesting GRE numbers. You remember the post where I analyzed and reported on GRE scores by intended graduate school concentration? It was a very popular post (for example, philosophy departments like it because it highlights that people who want to study philosophy have very strong GRE scores).

As it happens the table which I reported on is relatively coarse. ETS has a much more fine-grained set of results. Want to know how aspiring geneticists stack up against aspiring ecologists? Look no further! There are a lot of disciplines. I wanted to focus on the ones of interest to me, and I limited them to cases where the N was 100 or greater (though many of these have N’s in the thousands).

You’re going to have to click the image to make out where the different disciplines are. But wait! First I need to tell you what I did. I looked at the average verbal and mathematical score for each discipline. Then I converted them to standard deviation units away from the mean. This is useful because there’s an unfortunate compression and inflation on the mathematical scores. Disciplines which are stronger in math are going to have a greater average because the math averages are higher all around. You can see that I divided the chart into quadrants. There are no great surprises. People who want to pursue a doctorate in physical education are in the bottom left quadrant. Sorry. As in my previous post physicists, economists, and philosophers do rather well. But there were some surprises at the more detailed scale. Historians of science, and those graduate students who wish to pursue classics or classical languages are very bright. Budding historians of science have a relatively balanced intellectual profile, and the strongest writing scores of any group except for philosophers. I think I know why: many of these individuals have a science background, but later became interested in history. They are by nature relatively broad generalists. I have no idea why people drawn to traditionally classical fields are bright, but I wonder if it is because these are not “sexy” domains, to the point where you have to have a proactive interest in the intellectual enterprise.

I also wanted to compare aggregate smarts to intellectual balance. In the plot to the right on the x-axis you have the combined value of math and verbal scores in standard deviation units. A negative value indicates lower values combined, and a positive value higher. Obviously though you can have a case where two disciplines have the same average, but the individual scores differ a lot. So I wanted to compare that with the difference between the two scores. You can see then in the plot that disciplines like classics are much more verbal, while engineering is more mathematical. Physical scientists tend to be more balanced and brighter than engineers. Interestingly linguists have a different profile than other social scientists, and cognitive psych people don’t cluster with others in their broader field. Economists are rather like duller physicists. Which makes sense since many economists are washed out or bored physicists. And political science and international relations people don’t stack up very well against the economists. Perhaps this is the source of the problem whereby economists think they’re smarter than they are? Some humility might be instilled if economics was always put in the same building as physics.

In regards to my own field of interest, the biological sciences, not too many surprises. As you should expect biologists are not as smart as physicists or chemists, but there seems to be two clusters, with a quant and verbal bias. This somewhat surprised me. I didn’t expect ecology to be more verbal than genetics! And much respect to the neuroscience people, they’re definitely the smartest biologists in this data set (unless you count biophysicists!). I think that points to the fact that neuroscience is sucking up a lot of talent right now.

The main caution I would offer is that converting to standard deviation units probably means that I underweighted the mathematical fields in their aptitudes, because such a large fraction max out at a perfect 800. That means you can’t get the full range of the distribution and impose an artificial ceiling. In any case, the raw data in the table below. SDU = standard deviation units.

 

Field V-mean M-mean V-SDU M-SDU Average-SDU Difference-SDU
Anatomy 443 568 -0.16 -0.11 -0.13 -0.05
Biochemistry 486 669 0.20 0.56 0.38 -0.36
Biology 477 606 0.13 0.15 0.14 -0.02
Biophysics 523 727 0.51 0.95 0.73 -0.43
Botany 513 626 0.43 0.28 0.35 0.15
Cell & Mol Bio 497 658 0.29 0.49 0.39 -0.20
Ecology 535 638 0.61 0.36 0.49 0.26
Develop Bio 490 623 0.24 0.26 0.25 -0.02
Entomology 505 606 0.36 0.15 0.25 0.22
Genetics 496 651 0.29 0.44 0.36 -0.16
Marine Biology 499 611 0.31 0.18 0.24 0.13
Microbiology 482 615 0.17 0.21 0.19 -0.04
Neuroscience 533 665 0.60 0.54 0.57 0.06
Nutrition 432 542 -0.25 -0.28 -0.27 0.03
Pathology 468 594 0.05 0.07 0.06 -0.02
Pharmacology 429 634 -0.28 0.33 0.03 -0.61
Physiology 464 606 0.02 0.15 0.08 -0.13
Toxicology 465 610 0.03 0.17 0.10 -0.15
Zoology 505 609 0.36 0.17 0.26 0.20
Other Biology 473 626 0.09 0.28 0.19 -0.19
Chemistry, Gen 483 681 0.18 0.64 0.41 -0.47
Chemistry, Analytical 464 652 0.02 0.45 0.23 -0.43
Chemistry, Inorganic 502 690 0.34 0.70 0.52 -0.37
Chemistry, Organic 490 683 0.24 0.66 0.45 -0.42
Chemistry, Pharm 429 647 -0.28 0.42 0.07 -0.69
Chemistry, Physical 513 708 0.43 0.82 0.62 -0.39
Chemistry, Other 477 659 0.13 0.50 0.31 -0.37
Computer Programming 407 681 -0.46 0.64 0.09 -1.10
Computer Science 453 702 -0.08 0.78 0.35 -0.86
Information Science 446 621 -0.13 0.25 0.06 -0.38
Atmospheric Science 490 673 0.24 0.59 0.41 -0.35
Environ Science 493 615 0.26 0.21 0.23 0.06
Geochemistry 514 657 0.44 0.48 0.46 -0.05
Geology 495 625 0.28 0.27 0.27 0.01
Geophysics 487 676 0.21 0.61 0.41 -0.40
Paleontology 531 621 0.58 0.25 0.41 0.33
Meteology 470 663 0.07 0.52 0.30 -0.46
Epidemiology 485 610 0.19 0.17 0.18 0.02
Immunology 492 662 0.25 0.52 0.38 -0.26
Nursing 452 531 -0.08 -0.35 -0.22 0.27
Actuarial Science 460 726 -0.02 0.94 0.46 -0.96
Applied Math 487 730 0.21 0.97 0.59 -0.76
Mathematics 523 740 0.51 1.03 0.77 -0.52
Probability & Stats 486 728 0.20 0.95 0.58 -0.75
Math, Other 474 715 0.10 0.87 0.48 -0.77
Astronomy 525 706 0.53 0.81 0.67 -0.28
Astrophysics 540 727 0.66 0.95 0.80 -0.29
Atomic Physics 522 739 0.50 1.03 0.77 -0.52
Nuclear Physicsl 506 715 0.37 0.87 0.62 -0.50
Optics 495 729 0.28 0.96 0.62 -0.68
Physics 540 743 0.66 1.05 0.85 -0.40
Planetary Science 545 694 0.70 0.73 0.71 -0.03
Solid State Physics 514 743 0.44 1.05 0.74 -0.62
Physics, Other 519 723 0.48 0.92 0.70 -0.44
Chemical Engineering 490 729 0.24 0.96 0.60 -0.72
Civil Engineering 456 705 -0.05 0.80 0.38 -0.85
Computer Engineering 465 716 0.03 0.87 0.45 -0.85
Electrical Engineering 465 722 0.03 0.91 0.47 -0.89
Industrial Engineering 426 699 -0.30 0.76 0.23 -1.06
Operations Research 483 743 0.18 1.05 0.61 -0.88
Materials Science 509 728 0.39 0.95 0.67 -0.56
Mechanical Engineering 471 721 0.08 0.91 0.49 -0.83
Aerospace Engineering 498 725 0.30 0.93 0.62 -0.63
Biomedical Engineering 504 717 0.35 0.88 0.62 -0.53
Nuclear Engineering 500 720 0.32 0.90 0.61 -0.58
Petroleum Engineering 414 676 -0.40 0.61 0.10 -1.01
Anthropology 532 562 0.59 -0.15 0.22 0.73
Economics 508 707 0.39 0.81 0.60 -0.43
International Relations 531 588 0.58 0.03 0.30 0.55
Political Science 523 574 0.51 -0.07 0.22 0.58
Clinical Psychology 484 554 0.18 -0.20 -0.01 0.38
Cognitive Psychology 532 627 0.59 0.28 0.44 0.30
Community Psychology 441 493 -0.18 -0.60 -0.39 0.43
Counseling Psychology 444 500 -0.15 -0.56 -0.35 0.41
Developmental Psychology 476 563 0.12 -0.14 -0.01 0.26
Psychology 476 546 0.12 -0.25 -0.07 0.37
Quantitative Psychology 515 629 0.45 0.30 0.37 0.15
Social Psychology 518 594 0.47 0.07 0.27 0.40
Sociology 490 541 0.24 -0.28 -0.02 0.52
Criminal Justice/Criminology 418 477 -0.37 -0.71 -0.54 0.34
Art history 536 549 0.62 -0.23 0.20 0.85
Music History 536 596 0.62 0.08 0.35 0.54
Drama 514 541 0.44 -0.28 0.08 0.72
Music History 490 559 0.24 -0.17 0.03 0.40
Creative Writing 553 540 0.76 -0.29 0.24 1.06
Classical Language 619 633 1.32 0.32 0.82 0.99
Russian 584 611 1.03 0.18 0.60 0.85
American History 533 541 0.60 -0.28 0.16 0.88
European History 554 555 0.77 -0.19 0.29 0.97
History of Science 596 661 1.13 0.51 0.82 0.62
Philosophy 591 630 1.08 0.30 0.69 0.78
Classics 609 616 1.24 0.21 0.72 1.02
Comp Lit 591 588 1.08 0.03 0.56 1.06
Linguistics 566 630 0.87 0.30 0.59 0.57
Elementary Education 438 520 -0.20 -0.42 -0.31 0.22
Early Childhood Education 420 497 -0.35 -0.58 -0.46 0.22
Secondary Education 484 576 0.18 -0.05 0.07 0.24
Special Education 424 497 -0.32 -0.58 -0.45 0.26
Physical Education 389 487 -0.61 -0.64 -0.63 0.03
Finance 466 721 0.03 0.91 0.47 -0.87
Business Adminstraiton 434 570 -0.24 -0.09 -0.16 -0.14
Communication 458 517 -0.03 -0.44 -0.24 0.41
Theology 537 583 0.63 -0.01 0.31 0.64
Social Work 428 463 -0.29 -0.80 -0.54 0.52

December 10, 2010

Verbal vs. mathematical aptitude in academics

It isn’t too difficult to find GRE scores by intended major online. In reviewing articles/posts for my post below on anthropology I noted the distinction made between quant & qual methods, and aversions to regressions and scatter plots (or the supposed love of biological anthropologists for these tools). That got me wondering about the average mathematical and verbal aptitudes of those who intend to pursue graduate work in anthropology. I removed some extraneous disciplines which I don’t think add anything, and naturally I created three scatter plots, quantitative score vs. verbal score, writing score vs. verbal score, and writing score vs. quantitative score.

I was more interested in the spatial relationships between disciplines. But, I was a but surprised by the low correlations between quant and verbal scores at the level of disciplines. On the individual level there’s naturally some correlation. People who score very high in one are unlikely to score very low in another. That’s why the variance in scores of a simple 10 word vocabulary test can predict 50% of the variance in general intelligence. In any case, here are the r-squareds:

quant-verbal = 0
writing-verbal = 0.81
writing-quant = 0.08

So 81% of the variance in writing scores on the scale of disciplines can be explained by verbal scores. Below are the three scatter plots:

gre1gre2gre3

Some observations:

- Social work people have more EQ than IQ (this is not a major achievement because of the scale obviously).

- Accountants never made it into the “blue bird” reading group.

- Philosophers are the smartest humanists, physicists the smartest scientists, economists the smartest social scientists.

- Yes, anthropologists can read and write far better than they can do math.

The raw data below.

Major Verbal Quant Writing
Philosophy 589 636 5.1
English 559 552 4.9
History 543 556 4.8
Art History 538 554 4.7
Religion 538 583 4.8
Physics 534 738 4.5
Anthropology 532 571 4.7
Foreign Language 529 573 4.6
Political Science 522 589 4.8
Economics 504 706 4.5
Math 502 733 4.4
Earth Science 495 637 4.4
Engineering, Materials 494 729 4.3
Biology 491 632 4.4
Art & Performance 489 571 4.3
Chemistry 487 682 4.4
Sociology 487 545 4.6
Education, Secondary 486 577 4.5
Engineering, Chemical 485 727 4.3
Architecture 477 614 4.3
Banking & Finance 476 709 4.3
Communications 470 533 4.5
Psychology 470 543 4.5
Computer Science 469 704 4.2
Engineering, Mechanical 467 723 4.2
Education, Higher 465 548 4.6
Agriculture 461 596 4.2
Engineering, Electrical 461 728 4.1
Engineering, Civil 457 702 4.2
Public Administration 452 513 4.3
Education, Elementary 443 527 4.3
Engineering, Industrial 440 710 4.1
Business Administration 439 562 4.2
Social Work 428 468 4.1
Accounting 415 595 3.9

Powered by WordPress