Old and New MCAT interpretation

[hope_2016]

There is a lot of discussion about old and new MCAT scores and how schools will look at it.
Why was MCAT changed in the first place? I think the idea was to capture a particular/different profile of applicants that would make more successful physicians. AAMC official guide recommends using new scores instead.
If that is true , it won't be logical to combine or average new and old score. You either look at the old or the new to make a decision. Thoughts?

---

Members don't see this ad.

 

[Goro]

We're averaging scores on any repeat, and using percentiles to compare the old and new exams. We're also not buying the rational that AAMC used to change the exam, either.

 

[Lawpy]

We're averaging scores on any repeat, and using percentiles to compare the old and new exams. We're also not buying the rational that AAMC used to change the exam, either.

Doesn't using percentiles require the assumption that both the old and new MCAT scores are normally distributed?

This also means the subsection scores in both exams are also normally distributed. And how would you factor in the new psych/soc section?

 

[hope_2016]

We're averaging scores on any repeat, and using percentiles to compare the old and new exams. We're also not buying the rational that AAMC used to change the exam, either.

It is interesting that academia is not buying what AAMC is trying to sell with the new exam and applicants are caught in this confusion.

 

[Goro]

Percentiles are percentiles.

Doesn't using percentiles require the assumption that both the old and new MCAT scores are normally distributed?

This also means the subsection scores in both exams are also normally distributed. And how would you factor in the new psych/soc section?

 

[Lawpy]

Percentiles are percentiles.

Yeah I understood. I'm just concerned that simple percentile comparison may not be accurate.

When I revisited the 2012-2014 AAMC MCAT data charts used in the @efle Percentile Tables, something looks odd:

• The mode for the total scores is 27, while the mean is 25.2. (median = 5oth percentile, so median is close to 25)
• The mode for the PS score is 7, while the mean is 8.3. (median = a little below 8)
• The mode for the VR score is 10, while the mean is 8.1. (median = a little below 8)
• The mode for the BS score is 10 (very sharply too), while the mean is 8.8. (median = a little below 9)

There's a significant discrepancy between the mode and the mean for the total and subsection scores (while the mean and median are more closely aligned). The histograms don't look normal, but instead rather skewed to a side.

In contrast, the May 2016-April 2017 AAMC Percentile Ranks seems more aligned:

• The mode for the total scores is 499, the mean is 499.6, the median is a little above 499
• The mode for the C/P score is 124, the mean is 124.9, the median is about 124.5*
• The mode for the CARS score is 125, the mean is 124.6, the median is roughly 124
• The mode for the B/B score is 125, the mean is 125, the median is about 124.5*
• The mode for the P/S score is 125, the mean is 125, the median is about 124.5*

*these medians are somewhat underestimated since 124 = 44th percentile, 125 = 55th percentile usually

So the new MCAT score histograms look nicely normal, whereas the old MCAT score histograms look rather skewed. So wouldn't the direct percentile comparison be inaccurate because we're dealing with percentiles from two different distributions?

 

[efle]

So wouldn't the direct percentile comparison be inaccurate because we're dealing with percentiles from two different distributions?

Nope. Percentiles are percentiles. It remains accurate to say that a 512 and 32 performed equivalent or better than the same fraction (88%) of test takers. You don't need a normal distribution to talk about percentiles.

Differences like the percentile bins being bigger or smaller between numerical scores might change the confidence intervals around that value, but the values themselves remain equal. So, while the old test was a 66% confidence interval of +/- 2, the new test might be more narrow, but a 518 vs 36 are still going to have the same rarity among applicants.

 

[Lawpy]

Nope. Percentiles are percentiles. It remains accurate to say that a 512 and 32 performed equivalent or better than the same fraction (88%) of test takers. You don't need a normal distribution to talk about percentiles.

Differences like the percentile bins being bigger or smaller between numerical scores might change the confidence intervals around that value, but the values themselves remain equal. So, while the old test was a 66% confidence interval of +/- 2, the new test might be more narrow, but a 518 vs 36 are still going to have the same rarity among applicants.

What? So regardless of the distributions involved, percentiles can just be mapped directly? So if i have an exam that has a score distribution being bimodal with a mean well above the expected (i.e. the middle value of the score scale), that can be adequately compared with an exam that's properly normalized using percentiles? That doesn't make sense, because the quality of two exams are different to make any reasonable assessment on the academic quality of the applicants.

In regards to the two MCATs, the means of both exams would show that the new MCAT is somewhat harder than the old MCAT. This could be due to a lack of familiarity (but it's a year the new exam is out). The modes also show that the largest fraction of test takers do well on bio/VR and poorly on PS. These show the problems that the old exam suffer, which the new exam seems to have resolved. That's why distributions matter when making percentile comparisons

 

[hope_2016]

Nope. Percentiles are percentiles. It remains accurate to say that a 512 and 32 performed equivalent or better than the same fraction (88%) of test takers. You don't need a normal distribution to talk about percentiles.

Differences like the percentile bins being bigger or smaller between numerical scores might change the confidence intervals around that value, but the values themselves remain equal. So, while the old test was a 66% confidence interval of +/- 2, the new test might be more narrow, but a 518 vs 36 are still going to have the same rarity among applicants.

Statistical arguments are all great, but the content was redesigned as well, and there was a more biological context to all the sections, and a whole new section on psychology/sociology, which in my opinion is a different exam targeting a different kind of applicant

 

[Lawpy]

Statistical arguments are all great, but the content was redesigned as well, and there was a more biological context to all the sections, and a whole new section on psychology/sociology, which in my opinion is a different exam targeting a different kind of applicant

The new MCAT is just an extension of the old exam to include P/S. Each section was extended by adding few more passages but both exams assess critical thinking/reasoning skills

 

[efle]

What? So regardless of the distributions involved, percentiles can just be mapped directly? So if i have an exam that has a score distribution being bimodal with a mean well above the expected (i.e. the middle value of the score scale), that can be adequately compared with an exam that's properly normalized using percentiles? That doesn't make sense, because the quality of two exams are different to make any reasonable assessment on the academic quality of the applicants.

In regards to the two MCATs, the means of both exams would show that the new MCAT is somewhat harder than the old MCAT. This could be due to a lack of familiarity (but it's a year the new exam is out). The modes also show that the largest fraction of test takers do well on bio/VR and poorly on PS. These show the problems that the old exam suffer, which the new exam seems to have resolved. That's why distributions matter when making percentile comparisons

Saying one exam is poorly normalized or invalid in general is fine. But it has nothing to do with percentiles. Distributions can be wildly different and both will have some score at which only X% met or exceeded.

the new MCAT is somewhat harder than the old MCAT

Percentiles. There is no easier/harder.

Statistical arguments are all great, but the content was redesigned as well, and there was a more biological context to all the sections, and a whole new section on psychology/sociology, which in my opinion is a different exam targeting a different kind of applicant

AAMC has been saying similarly, that you shouldn't try to compare between tests...but then again they also want admissions to treat 500 as a good enough score and stop emphasizing only the top fifth of the range. The admissions people on these boards have said they're comparing percentiles.

 

[Lawpy]

Saying one exam is poorly normalized or invalid in general is fine. But it has nothing to do with percentiles. Distributions can be wildly different and both will have some score at which only X% met or exceeded.


Percentiles. There is no easier/harder.


AAMC has been saying similarly, that you shouldn't try to compare between tests...but then again they also want admissions to treat 500 as a good enough score and stop emphasizing only the top fifth of the range. The admissions people on these boards have said they're comparing percentiles.

The difficulty was in regards to looking at means and modes relative to the expected midpoint of 24 (8 in each subsection).

How are percentiles calculated?

 

[efle]

The difficulty was in regards to looking at means and modes relative to the expected midpoint of 24 (8 in each subsection).

How are percentiles calculated?

The scaled values are arbitrary, eg the fact that more people got a 10 than 9 in PS does not give you any information about difficulty, it's just a result of where the bins they chose begin and end.

Take the number of people performing as well or worse, and divide by the total number of test takers.

 

[MADD!!!]

One reason I heard the AAMC changed the exam is because too many people were starting to score 30+ on the old MCAT, therefore the MCAT was not doing it's intended job of screening out applicants as well. With the new exam, making it more difficult and adding more actual scores someone can get helps to shake up the exam so it produces a more normalized distribution. This will allow the AdComs an easier time to distinguish between applicants.

 

[jesie]

One reason I heard the AAMC changed the exam is because too many people were starting to score 30+ on the old MCAT, therefore the MCAT was not doing it's intended job of screening out applicants as well. With the new exam, making it more difficult and adding more actual scores someone can get helps to shake up the exam so it produces a more normalized distribution. This will allow the AdComs an easier time to distinguish between applicants.

It's job is not to screen out...it's to evaluate (the validity of that validation is up to the judgement of the individual I'm not making any claims on that). Schools can then use that evaluation to do screening, but the aamc does not seem to be intent on screening. Your point about 'too many people getting 30+" as a reason to change the test is silly. A certain percentage will score that high. The only reason you'd get 'too many 30+' is because you have too many people taking the test. It's all on a curve, bud, which is why 'making it harder' is an arbitrary statement...that would only adjust the raw scores and would have no influence on the percentiles, since everyone would be brought down by it.


Sent from my iPhone using SDN mobile