Best Fitting line in testing

[spillsomepaint]

---

Members don't see this ad.

I had a discussion with a professor today about the discrepancies in standardized tests like the GRE. She suggested that ETS should have different scoring methods for groups that the tests are biased against. Since the bias isn't related to content or face validity, scoring the tests different for the various ethnic groups, gender, or even age, might make the GRE's more valid and reliable at predicting graduate school success. Thoughts?

 

[JockNerd]

I had a discussion with a professor today about the discrepancies in standardized tests like the GRE. She suggested that ETS should have different scoring methods for groups that the tests are biased against. Since the bias isn't related to content or face validity, scoring the tests different for the various ethnic groups, gender, or even age, might make the GRE's more valid and reliable at predicting graduate school success. Thoughts?

There was a thread on this a while ago (I've been writing that a lot lately😛)

Doing analysis based on categorical variables like ethnicity when better alternatives are available is a mistake, methodologically speaking. It's similar to, say, making an arbitrary distinction at the mean of a test and testing for differences between the "high" and "low" group. It obscures useful information and introduces a much less meaningful categorization.

The variable you and your prof are interested in isn't ethnicity, to take one of your examples. The variables you're interested in are english language proficiency, acculturation maybe, SES maybe, format of education in the person's country of origin, etc.

Say we made score adjustments. If adjustments were made on a broad categorical variable just as ethnicity, here's what would happen: The most affluent of the minority population, those least in need of the adjustment, will have their scores adjusted anyhow. These individuals would reap all the benefit from the adjustment while those whom it should target (say, a person from a foreign country who doesn't know a word of English, but can build a particle accelerator) will get nothing. Obviously that's a bit hyperbolic, but that's the basic process that would result.

Frankly, ETS is not responsible for fixing education. The time for social change is not after people write the GREs. The people who could have used assistance were left in the dust long before they got to that point. Saying that we should score the GRE differently for different ethnic groups is a weak, ineffective bandage for a pervasive problem in North America and the world.

 

[chaos]

Heck no.

If they make 'allowances' in scoring for some disadvantaged group, it would only be fair to adjust scores for every possible contingency-- learning disabilities, physical disabilities, race, gender, socioeconomic status. Plus, that is not even taking into consideration the reason *why* these groups perform poorly. Socioeconomic status probably plays somewhat of a role in race, but not really in gender...that's probably more of a cultural thing. I don't know what groups fall short when SES is controlled for, but the fact is, the GRE is a test of how well you know English and basic math skills you will need in grad school (ok, it is a horrible test that seems like it doesn't test any of those things but that's another topic). When you level out the scores, the results become pretty meaningless, don't they? Granted, the GRE doesn't have a whole lot of ecological validity (they say scores are predictive of success, but so are many other things, and mostly people on the high end of the bell curve get accepted anyway)...but say someone's score is 'adjusted' from 570 to say, 610 based on the fact that they are from a group that tends to perform poorly (statistically, without even addressing whether the construct being controlled for is the one producing the variability). If the grad school does not know about this adjustment, they might assume they are getting someone better versed in statistics or the English language than they really are. Of course, interviews would reveal potential weaknesses, and this adjustment might result in schools accepting students for interview that would not have been considered (well, the score difference would probably not be that significant, but say it was). Good news for them, bad news for the applicant that might have gotten edged out. And this would create the whole affirmative action backlash of students resenting other students, feeling they got in based on subsidy rather than merit.

I am just opposed to the idea of two people who answered a set of questions identically ending up with different scores. There are numerous other portions of the application where a student who may perform below average on the GRE can explain these mitigating factors, and show that they are stellar performers in other areas. But your GRE scores, in my opinion, should reflect the number of questions you answered correctly, and nothing more.

 

[deleted176373]

I had a discussion with a professor today about the discrepancies in standardized tests like the GRE. She suggested that ETS should have different scoring methods for groups that the tests are biased against. Since the bias isn't related to content or face validity, scoring the tests different for the various ethnic groups, gender, or even age, might make the GRE's more valid and reliable at predicting graduate school success. Thoughts?

It's not the test that is biased... However if I knew that I would be getting an easier test by checking the random ethnic block, you think that there would not be "cheating?"

Furthermore, what about Asian Americans, who on average score higher? Should we punish them (after all academia treats them as not "real" minorities to begin with.)

Maybe, just maybe, the bias is against how well they fair in graduate school education. Are you stating the irregardless of GRE score once you control for ethnicity, gender and age that the relationship between GRE & GPA to predict graduate school success goes away. Please cite me a study that shows this.

As an older student, I don't feel discriminated against by the GRE. I worked on what I needed to know and scored just fine on it. My wife, gasp, a poor female, outscored me by 140 points. I don't think she saw any bias either and she's an older student and female. To make matters worse she took it cold without preparation after finishing an online degree. Nope, her 1440 was not biased simply because she was white... it was simply that she's smart as hell. I studied for six months to pull a 1300, and no, I am not bitter about that.

The test is biased towards smart people with a high IQ and a great vocabulary.

Mark

PS - By older, she was 35 when she took the test and I was 38.

 

[spillsomepaint]

Maybe, just maybe, the bias is against how well they fair in graduate school education. Are you stating the irregardless of GRE score once you control for ethnicity, gender and age that the relationship between GRE & GPA to predict graduate school success goes away. Please cite me a study that shows this.
38.

No sources. Just a discussion I had with a professor and her opinions on the matter, which is not how I feel, but just something to think about.

 

[JockNerd]

No sources. Just a discussion I had with a professor and her opinions on the matter, which is not how I feel, but just something to think about.

There's actually another bias in your prof's thinking that occurred to me this morning.

What does the prof propose we do with scores, specifically, I wonder? My suspicion is that she wants, say, the marks of Asian Americans on verbal to be scaled down, the marks of, I don't know, some other ethnic group's math to be scaled up.... Ok, scaled up against what? The implicit assumption here is that white people are normative. I strongly suspect that she assumed norming would be done against how other groups fare compared to Whites. Hooray modern racism....

 

[Therapist4Chnge]

My arguments against this are the same for affirmative action (as it exists today), there is no 'fair' way to do it. I think the goal is to find a representative test, but we have fallen short in providing that. I think it is a necessary evil, but that doesn't means it should count for more than a data point amongst many data points. I think floor scoring can be helpful, though what is to say that a school won't continue to raise the floor score and cut out students the same way they would if they gave more emphasis on the scoring. In the end I think we need to de-emphasize the importance of the GRE, and instead focus on the other parts of the application that seem more appropriate to our field of study.

-t

 

[deleted176373]

My arguments against this are the same for affirmative action (as it exists today), there is no 'fair' way to do it. I think the goal is to find a representative test, but we have fallen short in providing that. I think it is a necessary evil, but that doesn't means it should count for more than a data point amongst many data points. I think floor scoring can be helpful, though what is to say that a school won't continue to raise the floor score and cut out students the same way they would if they gave more emphasis on the scoring. In the end I think we need to de-emphasize the importance of the GRE, and instead focus on the other parts of the application that seem more appropriate to our field of study.

-t

You make an interesting point about de-emphasis of the GRE, after all, who is really emphasizing this test?

I would argue that no program is required to emphasize or even require this test, certainly many professional programs do not. On the other hand, the very competitive Ph.D. Boulder model programs do. Why? and is that reasoning wrong?

I think, that the GRE is primarily used by programs along with GPA as a very gross sorting machine. Many may accuse me of a little elitism here, but there is a reason that some of us enter into doctoral programs, some only attain a masters, and others simply stop at a bachelors. No, it's not always due to ability, life gets in the way too.

Think about this, you have athletes every year who go from high school, to college, to professionals. Much like academics, not everyone is cut out to be a professional athlete, concert musician, or a medical doctor. There are a number of things blocking us all from achieving these things. People who go for their Ph.D./Psy.D. in clinical and counseling psychology are cut from a different cloth and not everyone is suited for it. Like professional athletes, we are at the top of the game, and like it or not we are competing to show that we belong there. Making the cut is different in our profession, instead of being able to run a 4 second 40, we have to have a 1200 GRE. It's a measure of our capability but also our dedication to making it into this field, just as doing hours of research and volunteer clinical experience are. No one hands any professional, be it football player or psychologist, anything. We are expected to work for it and I would suggest that a good number of us work very hard to get where we are. My GRE score was not the product of ability, it was the product of ability and desire.

Mark

 

[BorntoRun]

My GRE score was not the product of ability, it was the product of ability and desire.

I'll second that- I only got the GRE scores I did after months of really intense studying. Had I rolled out of bed that morning without studying, I would've bombed them. That said, there are people who do just that and ace the GREs. I think this translates in some ways to academia; there are people who succeed on sheer intellect, and there are people who succeed by sheer force of will- and it's very hard to tell the difference between these groups based on GRE scores- another reason the GREs may be less informative than they are considered to be by admissions committees.

PS: That's not to say that schools prefer sheer intellect over sheer force of will, or the other way around. I could see compelling arguments for wanting to recruit students with either of these characteristics, or ideally, with both. However, the GRE doesn't tell you what kind of student you're getting; I think LORs are the only way for schools to really learn about that. Not that this necessarily ties closely into the A/A discussion about GRE scores, but I think it represents another reason the GREs may be less predictive of 'success' in the first few years of graduate school than they are typically considered to be.

 

[Therapist4Chnge]

I would argue that no program is required to emphasize or even require this test, certainly many professional programs do not. On the other hand, the very competitive Ph.D. Boulder model programs do. Why? and is that reasoning wrong?

I only know of one school (The Wright Institute) that doesn't use the GRE as a ranking factor (though they require it for stats purposes).

As for why competitive programs (both Psy.D. and Ph.D.) emphasize the GRE more...it is because they can. It is just another way to differentiate potential candidates. I think it is limiting because I believe there are better factors to judge an applicant (GPA, Research, LOR, experience, etc).

-t