AMHERST, Mass. – It has been long debated whether the Graduate Record Examinations (GREs) are an appropriate selection tool for graduate school admissions, and whether overreliance on GRE scores may compromise admission of students historically underrepresented in science, technology, engineering and math (STEM) fields. Despite many concerns about the test, there have been few studies examining the efficacy of the GRE in predicting doctoral degree completion and even fewer examining this question in STEM fields.
In a new study of U.S./Permanent Resident students at state flagship research institutions published today in the journal PLOS ONE, a team of researchers led by Sandra L. Petersen of the University of Massachusetts Amherst has found convincing evidence that GRE scores are not predictive of STEM doctoral degree completion, and that relying on scores from the quantitative section (GRE Q) of the exam is likely to exclude talented students who score below arbitrarily defined “acceptable” scores, but who have other characteristics that are likely better predictors of success.
“Even though the Educational Testing Service (the organization that designs and administers the GRE) states that the GRE scores should not be used as a ‘cut-off’ for admission, this is a common practice,” Petersen continues. “Our study shows convincingly that the scores are not useful for identifying students most likely to finish STEM doctoral programs. In fact, the scores are negative predictors of completion rates for men.”
Petersen and her colleagues examined data from four state flagship research universities with graduate student enrollments ranging from approximately 1,500 to 14,000 that provided information for all U.S. citizens and permanent residents entering their STEM doctoral programs between 2000-05. The sample included 1,805 students, of whom 57.5 percent were men and 42.5 percent were women.
While they found no significant differences in GRE verbal (GRE V) scores between men and women, men had significantly higher GRE Q scores than women at every institution. Nevertheless, there were no gender differences in doctoral degree completion rates or the time to degree completion.
For women, completion rates were around 60 percent regardless of the quartile ranking of their scores. Surprisingly, the doctoral completion rates for men were actually the inverse of what one might expect. Men who completed STEM doctoral degrees had significantly lower GRE Q scores than those who left their programs. In fact, men in the lowest quartile of GRE Q scores finished their degrees at a higher rate (74 percent) than their counterparts in all higher quartiles, including those in the highest quartile who completed at a rate of just 56.2 percent. This pattern of ascending completion rates correlated with decreasing GRE scores was consistent among male cohorts in each institution studied.
“We hypothesized that the scores might have predictive power in engineering programs that require math skills, but even in those programs, that did not prove to be the case,” Petersen says. “Men in engineering doctoral programs who ranked in the lowest quartile of GRE Q scores – an average of 597 – finished their degrees at higher rates than those in the highest quartile with an average GRE Q score of 793.”
“The GRE scores might be meaningful if they predicted who would take longer to complete programs or who would leave after the first year because they failed coursework. But our data show that neither of these is true,” she says.
There is a potentially significant financial impact to these findings as well, the authors argue. The annual cost of training doctoral students in the four institutions studied averaged $58,000 per student, so when the researchers extrapolate those figures nationally they find the overall “dropout” cost is between $1-3 billion per incoming doctoral degree cohort, though this total does not include the potential value of papers, patents and contributions to the teaching mission created by doctoral students before leaving their programs.
“In view of this great cost to universities, not to mention personal costs to students and their families, why do we keep using a test that has not been found to be predictive of the endpoint we are trying to achieve?” Petersen asks. “One idea is that most of us in academia today necessarily scored highly to get into STEM doctoral training programs. Also, there are fewer examples of successful students with low scores if most of those chosen have high GRE scores. Finally, it has to be acknowledged that using GRE ‘cut-off’ scores speeds up the admissions process, particularly for programs that have high numbers of applications.”
“One reason given for using GRE scores to compare students is to ‘level the playing field’ for students coming from undergraduate institutions differing in prestige,” Petersen explains. “It has been suggested that without these scores, admissions committees may show implicit bias that could hurt the chances of admitting students from underrepresented groups who often come from lesser-known institutions. However, for undetermined reasons women and non-Asian minorities continue to score less well than white males and Asian Americans and, therefore, the pool of ‘acceptable’ women and minority candidates is reduced substantially. This is a looming problem because women comprise over 50 percent of the population, but earn only 25 percent of STEM doctoral degrees. Similarly, non-Asian minority groups currently comprise nearly 33 percent of the population, but earn only around 9 percent of the STEM doctoral degrees. If we are to remain a nation preeminent in STEM fields, we need to engage all segments of our population.”
Based on their data, Petersen and her colleagues believe that addressing the issue of GRE over-reliance may be a key to opening more doors to untapped talent, particularly underrepresented groups in STEM fields.
“One place to start may be identifying the characteristics that motivated admissions committees to overlook GRE scores of male students in the lowest quartile,” Petersen says. “We could then study whether these characteristics contributed to the high completion rates of this group of students, and in doing so, perhaps we can develop a more inclusive and predictive STEM doctoral admissions processes.”
The team says that while policy makers at the National Science Foundation, National Institutes of Health and other funding agencies have removed GRE scores from their decision-making policies, admissions committees who control the selection of students have been slower to give up using the GRE scores.
“Most scientists and engineers argue forcefully that we need to use data to direct research and policy,” Petersen says. “If we truly believe this, it seems even more important to incorporate the present findings into the process that selects students who will actually conduct the research.”
Petersen, a professor of neuroendocrine studies in the department of veterinary and animal sciences at UMass Amherst, led the study as part of her role as director of the Northeast Alliance for Graduate Education and the Professoriate (NEAGEP). Formed in 1999 through a NSF grant, the NEAGEP is a 15-institution alliance of faculty and administrators with a shared goal of developing new initiatives to help diversify the STEM workforce. She was joined in the research project by Evelyn S. Erenrich, associate graduate dean of Rutgers University, Dovev L. Levine, assistant graduate dean of the University of New Hampshire, Jim Vigoreaux, associate provost for faculty affairs at the University of Vermont, and Krista Gile, associate professor of mathematics and statistics at UMass Amherst.
The full report, “Multi-institutional study of GRE scores as predictors of STEM Doctoral degree completion: GRE gets a low mark,” can be found here.