- Joined
- May 25, 2007
- Messages
- 4,119
- Reaction score
- 962
I figured with all the anxieties caused by the variability in test difficulties and topics covered on different sittings for the MCAT, a dilemma we have encountered might shine some light on what AAMC is probably encountering.
First, a little history on how we generated our CBT scales. We started with a pool of passages that we used before in the paper format for which we have a large N-value and thus a solid curve. On each of our CBTs, we placed three to five of these passages per section as well as some free-standing questions. When our students took the CBT exams, we collected question-by-question results and generated a curve based solely on those previously scored questions. We then compared this curve with the curve for the EXACT same questions from the paper format. It wound up that people were doing notably worse on the CBT version than the paper version (about 7% lower in terms of a raw score). We formulated all sorts of theories, but many came back to the fact that (1) people work more slowly on a computer than on paper and (2) not being able to write next to the question hurts one's performance if they cannot do it in their head.
From there, we then looked at the curve for the new questions and passages for which we had no previous data. Granted, these questions and passages were a bit more obscure than our field tested questions and passages, so we expected that the results would have been different. What we found was that the curve on these was pretty dang close to the field tested curve. The sequence of the exam was altered (just once so far) to see if that had an impact. I can definitely say that the sequence of the passages impacts one's performance, which supports the notion that ongoing anxieties play a crucial role in one's MCAT score.
By the time we got done analyzing the data, we realized that there were so many factors involved (from how much a student had reviewed to the time of day someone took the exam.) There are so many different factors to analyze that it's to the point of diminishing returns. So we picked a date to release our curve and basically said "WTF?, here it is. In many cases, the curve we got after much analysis was very close to the paper curve scaled down for the reduced number of questions and then shifted down by 7%. The bottom line is that you can curve anything.
So now for the present:
What has amazed us this year is that after getting scores back from our students who took the CBT in April, May, and June, is that they are getting a higher average score (by almost a full point), but the range is larger. We typically used to see a range of about 20 to 42 with an average of 30 to 31. Now we are seeing a range of 18 to 42, but with an average of 30.9 to 31.8. We modified our course to incorporate some CBT test-taking component, but I honestly believe that the increase is likely do to exposure to about two WTF type of weird passages on our practice exams. Freaking out during practice exams hopefully lightens the impact on the real exam.
So here is my hairbrained conclusion based soley on speculation. What I assume is happening is that the test is curved like before with various non-field tested questions being tossed out of the data pool after post-exam scrutiny. They analyze the data six ways to Sunday and realize it fits a Bell no matter how you cut it. Each separate sitting gets the same standardized results. So they are pleased with their curves. But the variability is not being accounted for, probably because they have no way of evaluating that. Unless test-takers take the exam multiple times, they have no way of seeing the impact from date to date. The reality is that with so few questions that they can't cover all of the topics, one exam is going to be better suited than another to a test taker's strengths.
You see it from practice exam to practice exam, with the fluctuation of your score based on the topics and how you are feeling that day. But AAMC does not see that variation; they just see Bell curve after Bell curve.
All you can really do is imporve your game plan for this exam. Learn from what others have encountered. The test is going to have some weird crap on it, and you have to figure out how to not freak out. Realize that some questions can be answered even when you don't really understand what it is asking. Employ POE (process of elimination) to the answers and you'll be fine in many instances.
I apologize to the few of you here who took our class and have heard some of these mantas/rants chanted over and over. But it's good to know the logic behind the suggestions.
First, a little history on how we generated our CBT scales. We started with a pool of passages that we used before in the paper format for which we have a large N-value and thus a solid curve. On each of our CBTs, we placed three to five of these passages per section as well as some free-standing questions. When our students took the CBT exams, we collected question-by-question results and generated a curve based solely on those previously scored questions. We then compared this curve with the curve for the EXACT same questions from the paper format. It wound up that people were doing notably worse on the CBT version than the paper version (about 7% lower in terms of a raw score). We formulated all sorts of theories, but many came back to the fact that (1) people work more slowly on a computer than on paper and (2) not being able to write next to the question hurts one's performance if they cannot do it in their head.
From there, we then looked at the curve for the new questions and passages for which we had no previous data. Granted, these questions and passages were a bit more obscure than our field tested questions and passages, so we expected that the results would have been different. What we found was that the curve on these was pretty dang close to the field tested curve. The sequence of the exam was altered (just once so far) to see if that had an impact. I can definitely say that the sequence of the passages impacts one's performance, which supports the notion that ongoing anxieties play a crucial role in one's MCAT score.
By the time we got done analyzing the data, we realized that there were so many factors involved (from how much a student had reviewed to the time of day someone took the exam.) There are so many different factors to analyze that it's to the point of diminishing returns. So we picked a date to release our curve and basically said "WTF?, here it is. In many cases, the curve we got after much analysis was very close to the paper curve scaled down for the reduced number of questions and then shifted down by 7%. The bottom line is that you can curve anything.
So now for the present:
What has amazed us this year is that after getting scores back from our students who took the CBT in April, May, and June, is that they are getting a higher average score (by almost a full point), but the range is larger. We typically used to see a range of about 20 to 42 with an average of 30 to 31. Now we are seeing a range of 18 to 42, but with an average of 30.9 to 31.8. We modified our course to incorporate some CBT test-taking component, but I honestly believe that the increase is likely do to exposure to about two WTF type of weird passages on our practice exams. Freaking out during practice exams hopefully lightens the impact on the real exam.
So here is my hairbrained conclusion based soley on speculation. What I assume is happening is that the test is curved like before with various non-field tested questions being tossed out of the data pool after post-exam scrutiny. They analyze the data six ways to Sunday and realize it fits a Bell no matter how you cut it. Each separate sitting gets the same standardized results. So they are pleased with their curves. But the variability is not being accounted for, probably because they have no way of evaluating that. Unless test-takers take the exam multiple times, they have no way of seeing the impact from date to date. The reality is that with so few questions that they can't cover all of the topics, one exam is going to be better suited than another to a test taker's strengths.
You see it from practice exam to practice exam, with the fluctuation of your score based on the topics and how you are feeling that day. But AAMC does not see that variation; they just see Bell curve after Bell curve.
All you can really do is imporve your game plan for this exam. Learn from what others have encountered. The test is going to have some weird crap on it, and you have to figure out how to not freak out. Realize that some questions can be answered even when you don't really understand what it is asking. Employ POE (process of elimination) to the answers and you'll be fine in many instances.
Get a good monitor for the real exam. If you get a bad one, demand a change.
If your first passage isn't easy at first glance (i.e., if it's a weird one), skip to the second passage. While this wastes a little time, it helps with the psyche.
Do as much as you can in your head, especially calculations. Visualize pictures and scribble minimalistic pictures when you need them. You get no partial credit for drawing quality.
Remember that the exam is curved. If a question is driving you crazy realize that it might be tossed out if everyone is randomly guessing. And if they keep it, the curve will be generous. When you get right down to it, the middle-difficulty questions and the difficult questions that can be analyzed using POE are the main components of the curve's spread. Do well on those and you'll do well on the exam.
The test has a great deal of randomness. If it doesn't go as you wish, there is a very good chance you will do better the next time. Learn from your experience.
I apologize to the few of you here who took our class and have heard some of these mantas/rants chanted over and over. But it's good to know the logic behind the suggestions.
