April or August MCAT "CURVE": Can it be truly STANDARDIZED?

This forum made possible through the generous support of SDN members, donors, and sponsors. Thank you.

zepplinfan

Senior Member
7+ Year Member
15+ Year Member
Joined
Dec 31, 2003
Messages
183
Reaction score
0
Quesiton:

The MCAT being a standardized test SHOULD be able to accurately gauge one's score regardless of the testing adminstration. I would assume that a 39 or a 30 on a test this year should reasonably correlate to a 39 or 30 on a test from the previous years.

However, clearly the MAJORITY of people taking the August MCAT would have a substantial difference in the OPPORTUNITY to study given classes and responsbilities during the school year. Whether or not people take advantage of this time is a different story. If we assume that a significant percent do take this time and ARE better prepared (ie. can only correlate to more knowledge, not neccisarily better test takers), one should expect a better score on the August MCAT than if those people take the MCAT in April. For those of you who say that knowledge does not matter, I refer you to the MSAR >

"The MCAT is also a test of achievement, not aptitude; the MCAT measures mastery of what one has learned rather than one's potential for learning.....review of coure material prior to taking the MCAT is highly recommended"

SO, let's say for the moment that better preperation leads to a higher RAW score on the mcat for this percent that decides to take the MCAT in August. When I refer to "higher" I mean a better raw score than if they were to take the SAME test without AS MUCH preparation (meaning they had other responsibilities or whatever).

Now here's where I don't understand how things works?? Maybe somebody could help me. I assume there is a score distribution for the April test. Now GIVEN THAT there is more people scoring a higher RAW score (and don't tell me there isn't, the MCAT people strive to make both tests equally difficult), the RAW score distribution for the August would be different. IMPORTANT: WE have no way of knowing FOR SURE if this isn't the true case>>MCAT only releases the SCALED SCORE. Who knows what the RAW scores were.

I realize they have to reset all the tests to past tests. But what I don't understand is:

If a signficant group of test takers get a higher RAW score on the August MCAT. How can the MCAT people differentiate between this being the result of "better prepared" test takers or that the "test was easier" than the April MCAT or any other test adminstration for example. The simply couldn't match curves (meaning forcing a distribution) because this would be unfair to the better prepared students in August > since the "MCAT is a test of achievement; not aptitude" > it would be possible that a better prepared student (than an April student) would score in the same percentile as (the lower prepared student in April)

My main point here: Though MCAT writers do try to make the tests of equal difficulty >> the only true gauge is the performance of the testakers. If the testakers themselves are skewed (meaning maybe more of them covered a particular topic better because they had more time ie. optics) and therefore did better, then how do the testmakers know that their test wasn't easier OR that the test-takers just knew the topics better? I would also imagine that since less people take the August MCAT these matters would only be amplified.

I think finally this comes down to the myth about an August MCAT "curve" being tougher than the April MCAT. I don't buy it since if it were, then the MCAT would not be truly standardized. But I can't seem to figure out a logic behind how they COULD standardized without intentionally or unintentionally hurting some test takers? Which makes the myth regarding the "curves" that much more interesting. Is my entire reasoning irrelevant in that the raw scores just aren't as different??? How do we know this fact??? MCAT only gives us the scaled scores correct???? We have no way of know how they do this????

I realize I have made a lot of assumptions (ones I think are reasonable to some degree) and such but I think you get the general Idea. Please any thoughts/comments/or whatever would be great.

~ Zep

Members don't see this ad.
 
okay, im going to tell you a "secret," but dont entirely take my word for it.

last year around june when i was waiting for my score i was very scared and anxious. so i called and emailed like a ton of people at aamc and the people who score the mcat (not aamc. its a private company that works for aamc that specializes in scoring and stuff). what they told me straight out and what i filtered from what they were saying is that there is basically no curve. in other words, the raw score to scaled score conversion box you see in practice test 5R and 6R, and soon 7R, are basically what they use for every mcat administration. the curve is rarely different; really it doesnt matter how much people study, since we are talking statistics and large sample size, certain things are bound to happen. you will notice that those conversion tables for 5R and 6R are not much different. i think it was something like every 10 years the conversion/curve is changed.

so all that curve stuff is BS. its basically what you see on 5R and 6R. now you can trust me or not. i will try to look for those emails and you can see what you think.
 
here is what they told me through email. i hope im not going to get into trouble. but here is info directly from them and it should really help clear some myths. let me know your thoughts and if this helps...

"MCAT scores are based on the four parts of the MCAT exam: Verbal
Reasoning (VR), Physical Sciences(PS), Biological Sciences (BS), and the
Writing Sample (WS).

For VR, there are 65 questions, with a scoring range of 1-13.
For PS, there are 77 questions, with a scoring range of 1-15.
For BS, there are 77 questions, with a scoring range of 1-15.
For WS, there are 2 questions, with a scoring range of J-T.

Your performance is compared with that of the people who took the exam
in 1991. Fancy statistics (equating) allow us to build a chain of
different forms of the exam that go all the way back to 1991. The goal is
to make it where it doesn't matter to you (the examinee) which set of
items you see, or who else takes the exam at the same time you do. The
first would be a problem if we just scored it by how many items you
answer correctly (like a percent-correct score) and the latter would be a
problem if we just ranked you against the people who took the exam at
the same time (or in the same year) as you did.

This way, if you get an 8, you know you are scoring about the same as
the average examinee in 1991. Who cares, you say? The more relevant
info might be that the average BS score is currently running about 8.4 -
i.e., the avg examinee today knows a little more BS (try not to chuckle
when you read that!) than the avg one in 1991. To the extent that
schools compare you with the others in this year's applicant pool, you are
being ranked with respect to this year's group - but that's not how
the scoring is done."

in another email they wrote me...

"Different test forms within a single test administration generally vary
slightly in difficulty. To ensure that no examinee benefits or is
disadvantaged because of these differences in test form difficulty, we do
post-scoring equating which ensures that, not only from form to form,
but also from administration to administration, examinees are assessed
according to the same standard. By this process, we slide more difficult
and less difficult forms in different directions along our standard
scale, aligning new-form raw score percentiles with those of the standard,
to determine what raw score on each new form corresponds to each of the
scaled scores on the standard. In that way we minimize the potential
effects of differences in test form difficulty, and ensure that for all
MCAT administrations, examinees are assessed according to the same
standard."
 
Members don't see this ad :)
Torjan,

Here's my only problem with what these MCAT makers say:


1. How do they define difficult tests? Are they claiming that they KNOW that a particular form is more difficult inherently? OR is it according to the actual RAW score distributions resulting from that particular form?

Based on this, my initial question is still valid. We can say that the SAMPLING SIZE IS LARGE to our hearts content. But we do NOT know this nor the RAW distribution.

For example: If I were to take a test in Augst and let's say there were 7 test forms for this test (I'm not really sure how many there are) And let's say there were 28,000 sitting for this adminstration.........so 4,000 people are given the same form. Let's say that a significant portion of these people (we don't know that actual number and neither does MCAT, so it shouldn't matter what the actual number is) is more well prepared because it is in August and they do better on this exam <<<<< meanin the RAW score distributions are skewed somewhat >>>> what does this mean?

If the testmakers are using RAW scores to determine if a form was "easier".......wouldn't this move the form down on the scaled score, EVEN though the test takers had a more complete "mastery" of the skills and achievement the test was aimed at in the first place?

If the test makers intentionally design a test form that is more difficult..........then this would make no sense.


maybe my idea of them equating stuff, just doesn't work out.

Ah, whatever, maybe I'm just not getting it. Am I reading too much into this thing, or does anyone else get my idea?

Also, trojan, I hear what your saying, I'm sure the scaled scores are similar, it's just that sometimes you hear that a form WAS SOOOO DIFFICULT compartively that it makes the issue of a curve seem important.
 
zepplinfan,
Aside from the comments regarding the formulae used to standardize administrations, I dont know what else you can expect. No test can be "truly standardized" because not only are there variations in the test forms, but also the locations of administration (band practice?), testing conditions (some guy constantly coughing loudly or talking to himself), etc.

So no, no test can ever be truly standardized. The MCAT people try to get as close as possibly with their tried and true statistical analysis, but they can only do so much.

Either way, Im not quite sure what you're expecting from this thread.
 
yah, ur right, it was just late night bs.....I just didn't want to study MCAT anymore and was trying to justify taking it in August. I think I'm going to catch some shut eye now.
 
I think the assumption you are making about the August MCATers having higher raw scores isnt right.

You could say, well the people who choose to take the test in april are less intimidated by the test/preparation, so these are the ones who will potentially score higher. Or any other silly reason to make the two months not equal.

But the fact is not everyone who takes the mcat is not a college kid operating on this standard schedule. So I feel like there is no rhyme or reason to distinguish april mcaters from the august in terms of abilities.

BUT you are right in the fact that we will never know the exact method which a particular test/form was scaled, which definitely gets you thinking conspiracy. As a previous poster said, you can score an 8 Today, and be Average compared to someone who took the test in 91. Whereas a 8.4 would be needed to be truly average currently.

So this means that as time passes raw scores (not nec from one month or another) are increasing. People today are getting more bubbles right that they were ten years ago, which i can see. So what does this mean?

This has to mean that each individual question is becoming more and more of an issue in terms of scaling. And in talking about the CURRENT applicant market, this means that your scaled score is more arbitrary than every before, especially at the extremes I feel. i.e. one or two questions will separate a 10 from an 11 today, but it was probably 3 or 4 ( or who knows) how many for that same separation years ago. Sorry for the rant, does this make sense tho?
 
forget about the curve. just go and study!!!
 
Have you guys seen how stunningly beautiful the bell curve was for the composite scores from April and August 2003? It was nearly perfect. Whatever their formulae or standards the AAMC is using, I think think they're doing a damn good job when it comes to the distribution.
 
It is fairly standardized. To differentiate between the August and April tests, it is actually quite simple. On the April exam, lets say, getting a raw score of 80 will give you a 10 on the section. In August, if it is true that the test takers are better prepared, then you would need to get an 85 to get the same score of 10. Better prepared examinees get more questions right, which leads to a bigger curve.
 
I didn't see the bell curve for those two tests. can you link me to them, flash?
 
Originally posted by zepplinfan
Quesiton:

The MCAT being a standardized test SHOULD be able to accurately gauge one's score regardless of the testing adminstration. I would assume that a 39 or a 30 on a test this year should reasonably correlate to a 39 or 30 on a test from the previous years.

However, clearly the MAJORITY of people taking the August MCAT would have a substantial difference in the OPPORTUNITY to study given classes and responsbilities during the school year. Whether or not people take advantage of this time is a different story. If we assume that a significant percent do take this time and ARE better prepared (ie. can only correlate to more knowledge, not neccisarily better test takers), one should expect a better score on the August MCAT than if those people take the MCAT in April. For those of you who say that knowledge does not matter, I refer you to the MSAR >

"The MCAT is also a test of achievement, not aptitude; the MCAT measures mastery of what one has learned rather than one's potential for learning.....review of coure material prior to taking the MCAT is highly recommended"

SO, let's say for the moment that better preperation leads to a higher RAW score on the mcat for this percent that decides to take the MCAT in August. When I refer to "higher" I mean a better raw score than if they were to take the SAME test without AS MUCH preparation (meaning they had other responsibilities or whatever).

Now here's where I don't understand how things works?? Maybe somebody could help me. I assume there is a score distribution for the April test. Now GIVEN THAT there is more people scoring a higher RAW score (and don't tell me there isn't, the MCAT people strive to make both tests equally difficult), the RAW score distribution for the August would be different. IMPORTANT: WE have no way of knowing FOR SURE if this isn't the true case>>MCAT only releases the SCALED SCORE. Who knows what the RAW scores were.

I realize they have to reset all the tests to past tests. But what I don't understand is:

If a signficant group of test takers get a higher RAW score on the August MCAT. How can the MCAT people differentiate between this being the result of "better prepared" test takers or that the "test was easier" than the April MCAT or any other test adminstration for example. The simply couldn't match curves (meaning forcing a distribution) because this would be unfair to the better prepared students in August > since the "MCAT is a test of achievement; not aptitude" > it would be possible that a better prepared student (than an April student) would score in the same percentile as (the lower prepared student in April)

My main point here: Though MCAT writers do try to make the tests of equal difficulty >> the only true gauge is the performance of the testakers. If the testakers themselves are skewed (meaning maybe more of them covered a particular topic better because they had more time ie. optics) and therefore did better, then how do the testmakers know that their test wasn't easier OR that the test-takers just knew the topics better? I would also imagine that since less people take the August MCAT these matters would only be amplified.

I think finally this comes down to the myth about an August MCAT "curve" being tougher than the April MCAT. I don't buy it since if it were, then the MCAT would not be truly standardized. But I can't seem to figure out a logic behind how they COULD standardized without intentionally or unintentionally hurting some test takers? Which makes the myth regarding the "curves" that much more interesting. Is my entire reasoning irrelevant in that the raw scores just aren't as different??? How do we know this fact??? MCAT only gives us the scaled scores correct???? We have no way of know how they do this????

I realize I have made a lot of assumptions (ones I think are reasonable to some degree) and such but I think you get the general Idea. Please any thoughts/comments/or whatever would be great.

~ Zep

"Ulysses" by Zeb. :laugh:

CCW
 
I was told by those who taught my MCAT class (Princeton Rev) and a proctor of the test that they take your score and compare you to everyone who took the same test form in that geographical area on the same day. They then compare this to the same test form throughout the country. They basically make sure there are no freakish outliers and they using the pool of highest scores make the curve. OHH And TROJAN The verbal section is now 1-15, and you can actually get a 15.
 
The MCAT is curved, but it's curved BEFORE anyone actually takes that year's test. They curve it by assessing the level of difficulty of the questions, based on students' performance on previous tests. In other words, they use questions that they have tested in the past, and curve from there.
 
A lot of this stuff could be made so much simpler if they'd just switch the exam to a computerized format. It would also make it easier to make sure you get through all those damn questions!! Scores would be produced faster and registry exams that have been switched to this format from paper format have shown better standardization and better ability to tell the examinees abilities/knowledge of the material. That's just my two cents though.
 
for w22 and everyone elses clarification, my post above (with quotes) was written by the aamc and the people who grade the mcat in a couple emails to me. so, they wrote that you could only get a 13 on the VR section, not me, :).
 
Can I make a general comment?
WHO CARES????
I know pre-meds are smart, detail-oriented people with a relative dearth of social obligations leaving time avaible for this type of pointless, erroneous assumption-driven analysis, but good lord people. Who cares? Moreover, do you really think you discovered the glitch in MCAT scoring? Please. Let's just assume (rightly, this time) that AMCAS/the medical establishment has it right, and devote your time to actually doing well on the test. Not to sound like a jerk, but this argument is insane, and I am dumbfounded by the earnestness of some of the replies--isn't it more fun/productive to just be sarcastic and light-hearted, and move on?
 
Top