"Inflated" April 2006 MCAT

Mister Pie said:

Has anyone looked at these tables?

The average score for the April '06 administration of the MCAT was 25.6, much higher than the combined averages for the last three years (24.7, 24.6, and 24.7). Does this mean that the August '06 MCAT will be curved more harshly? How will scores from other administrations of the test be compared to this one, seeing as how the percentiles corresponding to the scores are now somewhat different?

BTW, I apologize for the use of the word "inflated" since some might interpret it to mean that those who scored well this time around didn't deserve it, but that is hardly what I mean. I'm sure part of the reason the scores were higher this time around was that there were some really smart, hard working individuals who took the test.

Click to expand...

You make a good point, I noticed that too. The percentile for my score is equal (exactly) to one full point below in the previous administrations. I don't know why this happened. Maybe, like you said, it indicates a "better" testing population....how would AAMC know that, anyway? I don't get it either, but like I said one point isn't too much. It's not like a 30 is now a 25 now.

They'd know that since all of the questions were tested as experimentals in previous administrations. Or maybe I'm full of ****...

Anyway, the only way to interpret lower percentile scores for the same scaled scores is that the testing population performed slightly better than previous populations. Seems odd though...

Lests55 said:

You make a good point, I noticed that too. The percentile for my score is equal (exactly) to one full point below in the previous administrations. I don't know why this happened. Maybe, like you said, it indicates a "better" testing population....how would AAMC know that, anyway? I don't get it either, but like I said one point isn't too much. It's not like a 30 is now a 25 now.

Click to expand...

See, rescaling to a lower mean would penalize the April test takers if they did in fact perform better than other populations. This, I imagine, is the reason the average varies from administration to administration at all, and why questions need to be standardized on other populations before entering an exam. The other way to scale the MCAT would be to analyze the performance of a population in isolation, and forcing a mean of 24. However, doing this would give no way to control for natural examinee group variability and any drift over time. There is a well-known phenomenon called the Flynn Effect, for example, which clearly shows that IQ scores have been rising year after year for many decades even after correcting for covariables. If something like the Flynn Effect was happening on the MCAT, the same level of performance would yield lower and lower scores over time. Since this is bad, a mechanism needs to exist to equate testing administrations- so that a 13 on one exam is truly the same as a 13 on another. I don't imagine therefore that the scores from the April 2006 exam are "inflated" in any way- and since the percentile differences are so small, the slight overabundance of high scores in this application cycle should not affect anyone's particular chances at admission.

Mister Pie said:

Yeah, personally I would've thought that they could just rescale it to a slightly lower mean.

My score also corresponds to a lower percentile now if put on the April '06 scale, though this affects pretty much everyone else who took another administration of the test. The paranoid premed in me is worried! Curse you April people for being so clever!

Click to expand...

The experimental questions are most certainly real, and AAMC admits this freely. The only thing they don't tell us is how many there are on a particular test (my guess would be around 15%).
As far as weighing each question, I imagine that their algorithm incorporates both the statistics from the "experimental" performance and those from the "real" performance.
You are absolutely right when you expect that scaled scores and percentiles should closely correspond, however the very need for scaled scores at all admits to occasional and necessary variability. Otherwise, why wouldn't we just get the percentiles?

What was perplexing to me was that the mean did shift quite a bit between the 2005 and the April 2006 exams. Considering the sample size, this seems unusual. If I had to guess at an explanation, I would say that the imminent elimination of the paper test has aberrantly concentrated the number of better-prepared people in one administration. Someone who felt confident about the MCAT but was in no rush to take it may have all of a sudden felt compelled to register. Multiply this by several thousand, and voila! Indeed, if you compare the sample sizes from April 2005 to April 2006, a large increase is evident: 28,700 took the test in April 2005; 31,000 took it in April 2006. Moreover, since the risk:benefit ratio of taking the familiar paper version is only beneficial to prepared examinees, unprepared test takers did not register in excess.

Mister Pie said:

Hmm.... well my thinking was that a score of X should generally always roughly correspond to a certain percentile, and an increase of 1 point on a test such as the MCAT is HUGE. I'm not disputing your knowledge of statistics as it seems pretty clear to me that you know far more than me. Though it seems to me that the only way they'd know that say, the April 06 group was more qualified than previous groups would be if the rumored "experimental passages" on the MCAT do exist and they scale the test against that.

Another possibility might be that the raw scores were all lumped a bit more closely together this time, and moving the cut-off for say a 12 and a 13 would have shifted the percentiles even more than AAMC would have liked so they stuck with what they had.

This is kind of interesting (albiet fairly pointless) to think about.

Click to expand...

The Flynn effect that you mentioned shouldn't have much of an effect; this is probably one of the reasons that MCAT scores expire after three years. Since AAMC can't guarantee that a 13 now will equal a 13 then, they ensure that all comparisons in terms of med school admissions will be relatively equal by putting a short expiration date on scores. In short, the testmakers do not have to worry about rising IQ scores.

I only brought up the Flynn Effect as an example of drift that must be corrected for. You're right of course that the three year validity of scores eliminates the danger of score depreciation.

odddodo said:

The Flynn effect that you mentioned shouldn't have much of an effect; this is probably one of the reasons that MCAT scores expire after three years. Since AAMC can't guarantee that a 13 now will equal a 13 then, they ensure that all comparisons in terms of med school admissions will be relatively equal by putting a short expiration date on scores. In short, the testmakers do not have to worry about rising IQ scores.

Click to expand...