"Experimental" passages... Question for MCAT pros

[TheFreakMD]

Hey guys, i recently heard about the fact that one passage in both the PS and BIO sections does not count towards the final score. Is there any truth to that and can anybody explain? thanks

---

Members don't see this ad.

 

[mongrel]

I don't think so. With the new shortened format that would be making for a big chunk of meaninglessness. And what would even be the reasoning behind this?

 

[AnEyeLikeMars]

I took the paper MCAT so I can't really speak about the CBT. We did have experimental passages. The purpose was to test passages and questions for future exams.

 

[Mark84]

I don't see how that would be feasible, if a passage has say, 4 or 5 questions, thats the difference between a 12 and a 15 because of how tight the scale is. I'm not sure exactly how that would work.

 

[AnEyeLikeMars]

I don't see how that would be feasible, if a passage has say, 4 or 5 questions, thats the difference between a 12 and a 15 because of how tight the scale is. I'm not sure exactly how that would work.

It just means that that passage and questions are not used when converting raw scores to standardized scores. If they can created a distribution with 77 questions, why not 72? There are fewer questions with the CBT, but the same principle still applies.

 

[Shrike]

Mars, I am curious as to how you can be so sure about the experimental passages on your test. Was it just rumor, or what your prep class teachers told you? We have a great deal of difficulty figuring out exactly what the answer is on this, as AAMC's official line doesn't address the issue and their unofficial statements vary.

Our best guess is that there have always been, and still are on the CBT, experimental questions; entire experimental passages is less clear.

To those who argue that they can't put in experimental stuff because the sample size would be too small: by shrinking to a sample size of 144 a test whose scores vary over a 42-point range (or, even worse, 40 questions for a 14-point range, in VR), on which a single point can easily make a great deal of difference, AAMC has already demonstrated its poor grasp of statistics, or (more likely) that it doesn't care that the sample size is too small and the test too short. Why should we expect them to eschew shortening it a few more questions, when it would make their jobs so much easier?

 

[Mark84]

^^^^Thats along the lines of what I was thinking.

 

[AnEyeLikeMars]

Mars, I am curious as to how you can be so sure about the experimental passages on your test. Was it just rumor, or what your prep class teachers told you? We have a great deal of difficulty figuring out exactly what the answer is on this, as AAMC's official line doesn't address the issue and their unofficial statements vary.

Our best guess is that there have always been, and still are on the CBT, experimental questions; entire experimental passages is less clear.

To those who argue that they can't put in experimental stuff because the sample size would be too small: by shrinking to a sample size of 144 a test whose scores vary over a 42-point range (or, even worse, 40 questions for a 14-point range, in VR), on which a single point can easily make a great deal of difference, AAMC has already demonstrated its poor grasp of statistics, or (more likely) that it doesn't care that the sample size is too small and the test too short. Why should we expect them to eschew shortening it a few more questions, when it would make their jobs so much easier?

Page 4 of AAMC's MCAT essentials (right above the Test Section table)

"Each multiple-choice section will include some experimental items that do not count toward your score."

 

[AnEyeLikeMars]

"A word of caution is in order here with
respect to scores at the extreme ends of
the scale. Drawing conclusions about
differences for examinees at extreme
ends of the MCAT score scale is likely to
be unwarranted. Differences in levels of
predicted medical school performance
for examinees scoring one scale point
apart at the extremes tend to be small.
For instance, suppose records were being
reviewed for two individuals, one with
an 11 in Physical Sciences and another
with a 14 in Physical Sciences. The ± 1
score bands for these two individuals
would not overlap, suggesting
preparation and performance differences
exist. Results from predictive validity
studies demonstrate, however, that the
two individuals would be expected to
perform similarly in the medical school
curriculum."

This is from the 2005 MCAT interpretive manual. They also talk about confidence intervals and the reliability of scores, etc. This should be a must read for all test-takers and Adcomms.

http://www.aamc.org/students/mcat/mcatinterpmanual05.pdf

 

[Shrike]

Sure, adcoms should keep the facts behind the stats in mind. (If they actually have to read what you quoted above to figure out the statistical significance of a one-point MCAT difference, they're not too thoughtful, but that's another discussion.) But are you sure they do?

As for the stated line on experimental "items" (which does not have to mean passages, and probably comprises mostly individual questions), it's not quite clear how to take what they say. It's probably true, particularly as we have corroborating evidence, but you certainly shouldn't take AAMC pronouncements as gospel. Do I have to recite the misstatements they've made about, say, CBT details recently? They have a history of giving out confusing, misleading, and flat out incorrect information.

 

[AnEyeLikeMars]

Sure, adcoms should keep the facts behind the stats in mind. (If they actually have to read what you quoted above to figure out the statistical significance of a one-point MCAT difference, they're not too thoughtful, but that's another discussion.) But are you sure they do?

They should keep it in mind. Do all of them? I don't know. I just thought it was interesting that according them there probably shouldn't be a distinction between a 14 and an 11 for admissions purposes.

As for the stated line on experimental "items" (which does not have to mean passages, and probably comprises mostly individual questions), it's not quite clear how to take what they say. It's probably true, particularly as we have corroborating evidence, but you certainly shouldn't take AAMC pronouncements as gospel. Do you have to recite the misstatements they've made about, say, CBT details recently? They have a history of giving out confusing, misleading, and flat out incorrect information.

I don't really know where that's coming from...because I don't think I was doing that at all. I always thought it was common knowledge that they have experimental questions on the MCAT. Plus, they explicitly state it. That's enough for me.

EDIT: Looking back, you were making a distinction between experimental passages with experimental questions vs. real passages with one or two experimental questions thrown in. I thought the OP was concerned with experimental material in general. I think that's where the confusion is coming from. My bad.

 

[Shrike]

True that their position on 11 versus 14 is interesting and a bit surprising (though they're not saying there isn't a difference, they're saying there isn't a difference in med school performance).

Re experimental stuff: again, you're right that it's common knowledge" that there are experimental questions. My points were: (1) common knowledge isn't always right regarding the MCAT, though in this case it probably is; and (2) the OP asked about experimental passages, not questions, and there's no clear reason to believe there are entire experimental passages. Nor, for matter, much reason to believe there aren't -- we just don't know.

My edit: aha, we've got it now. [see above]