Analysis: A Percentile-based Model for MCAT Scale Conversion

[Lawpy]

NOTE: This analysis is obsolete and is superseded by a simple calculator. Nonetheless, I kept the report as is just to see how a statistical analysis alternative to a regression can be done.

This report will address a new method of converting from the 2015 MCAT scale (or really any MCAT scale) into the old, 45-point scale. Superficially, it seems similar to my previous analysis on LizzyM scores, but I felt the model was of immediate importance (since people don't have to repeatedly consult AAMC Conversion Tables).

Spoiler: Here is the model

Note the model produces the old MCAT in the range of 7 to 43, not 0 to 45, since it involved percentile matching and histogram analyses of AAMC percentile data. I will likely update it when more AAMC percentile data for the new MCAT are out, but for most important values, the model works well

old MCAT = (45/70) * (new MCAT - 461.11) (MCAT Conversion Model)

MCAT > 506 --> rounded down
478 < MCAT <= 506 --> rounded up
MCAT < 478 --> rounded down

Percentile Curve

@efle previously provided useful AAMC conversion/comparison tables based on side-by-side percentile comparisons. When percentiles are plotted against the 2015 MCAT scores, we observe the following percentile curve.

Lawper's 5th/50th/95th Percentile Method

Generally, if we are modeling continuous distributions on data with bounded endpoints (like 0-45 or 472-528), we should stick with the so-called beta distribution. However, the methodology and application of the beta distribution is too complex for the general purposes, so we'll use simpler distributions that serve as unbounded approximations.

The key assumption involved is that the distribution of the MCAT scores resembles a normal distribution. From the AAMC April-May 2015 MCAT date, the reported MCAT mean and standard deviation are 500 and 10.6 respectively, but we will stick strictly with percentiles.

From the above percentile curve, the most linear part is marked between 5th and 95th percentiles. Keep in mind that even without the percentile curve attached, the 5th and 95th percentiles are useful benchmarks that represent the edges of the normal curve. Even in statistical inference tests, the standard p-value used for assessing significance is below 0.05 (but don't confuse percentiles with p-values).

The corresponding MCAT scores are 481 (5th percentile), 500 (50th percentile) and 516 (95th percentile). The MCAT spread is defined as 2 * (95th percentile - 5th percentile), and thus equal to 2 * (516 - 481) or 70.

Consequently, mapping the new MCAT scale to the old, 45-point MCAT scale can be determined by the following formula:

old MCAT = (45 / MCAT spread) * (new MCAT - lowest new MCAT + correction factor)

Or more simply:

old MCAT = (45/70) * (new MCAT - 472 + correction factor)

I added the correction factor because further analysis (credit to @efle for finding this) of the histograms comparing the old and new MCAT indicates that the 528 on the new MCAT does not correspond to a 45 on the old MCAT. Because 43-45 is virtually indistinguishable on the old MCAT as opposed to significantly more people scoring 526-527 on the new MCAT, the 528 on the new MCAT will actually correspond to a 43 on the old MCAT.

Setting the old MCAT = 43 and new MCAT = 528, the correction factor is equal to -98/9 or -10.889. The following model for comparing the 2015 MCAT to the old MCAT is the following.

old MCAT = (45/70) * (new MCAT - 461.11) (MCAT Conversion Model)

Rounding Methods

Rather than using the round function in Excel (which worsens the accuracy of the model), I decided to play around with the floor (rounding down to the nearest integer) and ceiling (rounding up to the nearest integer) functions. The most important number used is 506, since it serves as a useful benchmark for the lowest average MCAT score possible to be competitive for MD schools, although this may change based on more AAMC data. The so-called piecewise rounding occurs at the following intervals:

MCAT > 506 --> rounded down
478 < MCAT <= 506 --> rounded up
MCAT < 478 --> rounded down

This piecewise rounding ensures that the model's accuracy is better than that of a direct linear regression.

Data Analysis

Of course, another way to determine a formula for MCAT conversions is by plotting two MCAT scales based on AAMC Percentile Conversion Tables and carrying out a simple regression on Excel. But that's not fun and the derivation is way more complicated. Additionally, the two-sigma method only requires new MCAT scores for three percentiles, so plotting out all the data is unnecessary. Nonetheless, I decided to include both just for comparison and accuracy testing.

Applying piecewise rounding to a linear curve results in a graph of a series of steps.The R^2 for the model is about 0.996, so it fits very well to the plotted data and is superior to the linear regression model. Equally important is that the rounded off data is one point more than the actual reported MCAT equivalent in some cases where MCAT > 509, and is one point lower in some cases where MCAT < 504. But that is the consequence of trying to plot a linear model into a nonlinear data that looks linear.

LizzyM Scores

Since medical schools evaluate both the old and new MCAT based on percentile comparisons, the percentile-matching model is effective and even stronger than the previously reported score-conserving model. However, since strict matching was involved in analyzing AAMC data, the equivalent ranges from the model for the old MCAT are from 7 to 43 instead of 0 to 45 (the reported AAMC ranges are from 5 to 43). This means the LizzyM scores are shifted from 0 to 85 to 5 to 83.

Right now, I will keep the score-conserving model as it is until I get more AAMC MCAT percentile data sometime around the end of this year. But hopefully, this model is useful to you!

Let me know if you have any questions/concerns/complaints, as well as any problems you find.

---

Members don't see this ad.

 

[efle]

Just to clarify my comments about the high score end: It appears from the histograms that the AAMC collapsed 43-45 (which used to be a bin anyways from Verbal being reported as "13-15") into 528. So a 528 is the same as a 45 in that they are both 100.00%ile but a 527 is not a 44, a 526 is not a 43

 

[WedgeDawg]

Isn't that javascript calculator site the best thing ever?

 

[Lawpy]

Isn't that javascript calculator site the best thing ever?

It is! Lots of fun involved.

Just to clarify my comments about the high score end: It appears from the histograms that the AAMC collapsed 43-45 (which used to be a bin anyways from Verbal being reported as "13-15") into 528. So a 528 is the same as a 45 in that they are both 100.00%ile but a 527 is not a 44, a 526 is not a 43

Yeah I kept 528 = 43, because it was messing up the model and the regression.

 

[Flashfan]

I like any chart that makes 516 a 35 instead of a 34.

 

[Silence in ER]

Or that makes a 515 a 34 instead of a 33.

However, I have a q for Lawper, for marginal cases like 515 or so on, how did you determine whether to go up to 34 or down to 33.
Forgive me cause I'm not really a big stats person.

 

[Lawpy]

I like any chart that makes 516 a 35 instead of a 34.

Or that makes a 515 a 34 instead of a 33.

However, I have a q for Lawper, for marginal cases like 515 or so on, how did you determine whether to go up to 34 or down to 33.
Forgive me cause I'm not really a big stats person.

So the issue is that the exact AAMC percentile data for April/May test takers form a nonlinear curve, meaning that a linear fit will be slightly off for some values. Both the regression and percentile-based models mark a 515 as a 34.2-34.4 and a 516 as a 34.8-35.0, so when rounded off to the nearest whole number, we would get a 34 and 35 respectively.

I tried using ways to round down any values but that actually led to worse results. Of course, more data from the AAMC for 2015 test takers will provide some clarification that i can use to make the model smoother, but until then, feel free to enjoy an extra +1 pt from the model 😉

 

[Flashfan]

Unfortunately, many adcoms on here didn't evaluate 95% with 35, but rounded it down. Over the years, I am sure that 95% will have its own image in someone's mind. For now, I am just pleased that my state school like stats in my range and that at least a few other schools have judged me worthy of an interview.

 

[Lawpy]

So rather than using the round function in Excel, I decided to play around with the floor (rounding down to the nearest integer) and ceiling (rounding up to the nearest integer) functions. The so-called piecewise rounding occurs at the following intervals:

MCAT > 506 --> rounded down
478 < MCAT <= 506 --> rounded up
MCAT < 478 --> rounded down

The model used is adjustment-free, involving only the correction factor (so: old MCAT = (45/70) * (new MCAT - 461.1111))

And below is the chart of the results (R^2 = 0.996). The JavaScript calculator is updated to include the revised model

Even with some rounding adjustments, the model isn't a perfect fit, and some values are off by a point (516 = 35, not 34 for example). Of course, I can try to subdivide further with more rounding, but that makes the model more complicated and less generalizable, so I would rather avoid that. But hopefully, this should be good enough

 

[Lawpy]

Why is a 522 a 39 and not a 38? Do you think it'll likely shift one way or the other based on the updated percentile tables released later this year?

The updated tables are really the key here since June-September data can induce some scoring shifts. The model itself differs from the exact data by a point at certain values because the exact data form a nonlinear curve, so they can't be matched exactly with a linear fit no matter how much rounding is done

 

[Doudline]

I hate stats.

 

[Lawpy]

Updated the first post with all current/revised models and charts.

I hate stats.

I can understand your sentiment. But I just stuck with the simple normal distribution and did nothing fancy 😛

 

[EverStriving]

So I apologize in advance for being such a downer, but I see a few issues with this model:

1) There are minimum and maximum possible scores for the MCAT, so one can't assume without justification that a normal distribution is a good fit. A normal distribution does not have maximum and minimum values.

Each score above a standard deviation from the mean is determined by the "68-95-99.7" rule, so the 68th, 95th, and 99.7th percentiles are respectively 1, 2, and 3 sigmas from the average.

2) Actually, this rule refers not to percentiles, but to the percentage of people within that distance from the mean, either higher or lower. For the upper-end percentiles, it's really (84.1-97.7-99.9). If a normal distribution were a good fit, the predicted scores for +1, +2, and +3 sigma would be 510.6, 521.2, and 531.8 respectively (using reported std. dev. of 10.6). The actual values for those percentiles are ~511, ~519, and 525-528. Clearly while the first two are decent fits, the third is not. Even the +2 sigma percentile strays a little from the predicted value. This may be a reflection of the "squishing" necessary at the edges to allow for maximum and minimum scores, and draws into question the model's accuracy for scores near the minimum and maximum.

If you re-calculate your 2-sigma 'MCAT spread' with the correct percentile values for +2 and -2 sigma, you get a spread equal to 2*(519-479) = 80.

old MCAT = (45/80) * (new MCAT - correction)

Deriving the correction:

43 = (45/80) * (528 - correction) --> correction = 451.6

So the fixed model would be:

old MCAT = (45/80) * (new MCAT - 451.6)

MCAT > 506 --> rounded down
478 < MCAT <= 506 --> rounded up
MCAT < 478 --> rounded down

This piecewise rounding ensures that the model's accuracy is better than that of a direct linear regression.

3) This piece-wise rounding method seems a bit ad hoc. Can you justify this schema on any logical/mathematical grounds?

4) Finally, and most importantly, how is this model better than what we can already deduce from comparing percentile ranks? What value does it add?

 

[molarmom]

Super interesting - I LOVE STATS!!

 

[Lawpy]

So I apologize in advance for being such a downer, but I see a few issues with this model:

Sure no problem. Criticism is necessary to help identify flaws and weaknesses (and it provides a good prep just in the lucky event I present this to AAMC 😛)

1) There are minimum and maximum possible scores for the MCAT, so one can't assume without justification that a normal distribution is a good fit. A normal distribution does not have maximum and minimum values.

That is a good point, and as you can see, the model is worse off in the lower extreme (a 472 = 7, not a 0). While the MCAT scale is finite and bounded, I was most concerned with the values in the intermediate-high range (say anywhere above a 500, which is important in assessing medical school chances). I chose the normal distribution as a model for two key reasons:

1. I assumed that any standardized test scores are normally distributed (i.e. there should be a set average with people scoring above and below)
2. I was looking at the histograms of the AAMC MCAT score distributions, and I noticed they resembled the Gaussian bell curve

Generally, I would prefer using a distribution-free model, but that's too much of advanced stats. Compared to other continuous, statistical distributions, the normal distribution accomplishes both #1 and 2 above and is fairly conceptual.

2) Actually, this rule refers not to percentiles, but to the percentage of people within that distance from the mean, either higher or lower. For the upper-end percentiles, it's really (84.1-97.7-99.9).

Yeah that was my mistake, so I'll fix it up. The values of 0.68-0.95-0.997 were indeed bidirectional, so thanks for the heads up.

If a normal distribution were a good fit, the predicted scores for +1, +2, and +3 sigma would be 510.6, 521.2, and 531.8 respectively (using reported std. dev. of 10.6). The actual values for those percentiles are ~511, ~519, and 525-528. Clearly while the first two are decent fits, the third is not. Even the +2 sigma percentile strays a little from the predicted value. This may be a reflection of the "squishing" necessary at the edges to allow for maximum and minimum scores, and draws into question the model's accuracy for scores near the minimum and maximum.

If you re-calculate your 2-sigma 'MCAT spread' with the correct percentile values for +2 and -2 sigma, you get a spread equal to 2*(519-479) = 80.

old MCAT = (45/80) * (new MCAT - correction)

Deriving the correction:

43 = (45/80) * (528 - correction) --> correction = 451.6

So the fixed model would be:

old MCAT = (45/80) * (new MCAT - 451.6)

Good analysis. I will see how this one works in conjunction to the slightly fixed up approach.

3) This piece-wise rounding method seems a bit ad hoc. Can you justify this schema on any logical/mathematical grounds?

The most important model is one that is linear, which will result in decimal values. The piecewise rounding is used to ensure that important values aren't rounded off too high/too low, which straightfoward rounding misses.

4) Finally, and most importantly, how is this model better than what we can already deduce from comparing percentile ranks? What value does it add?

Rather than drafting up many tables and meticulously comparing two different MCAT scales based on percentiles, the model uses three statistically determined values to produce equivalent values in another scale. It's basically compressing all the tables into a simple equation or a small calculator.

Appreciate your help, analysis and inquiry!