Why does step 1/mcat take so long to score

[kasho11]

Why does it take so long to grade a multiple choice test? I understand there is the writing portion on the MCAT but that will be removed next year and I strongly believe it will still take them 3-4 to score it. Step 1 however is, from what I've been led to believe, purely multiple choice, but still takes 4 weeks. Why do you not instantly get your scores like with the GRE? Do they review tests manually also?

Even the worst and oldest computer programs would be able to do all that work relatively quickly, even with a raw score conversion, a mcat/step 1 score from an answered test form.

Any thoughts?

---

Members don't see this ad.

 

[GTJackets]

I believe that a lot of your score is how you did relative to everyone else. So they have to obtain all the data, analyze it, and then determine the score distributions.

 

[kasho11]

For step 1 that might be true, I have not done much research on it, but the MCAT is supposed to be based on a predetermined scale. Even so, with computers doing the work that wouldn't take much time at all...

 

[Catalystik]

After they see all the test scores they have to "norm" the scores so that they are equivalent to a test taken at another time. Also, if a disproportionate number miss an item, they have to evaluate whether the question was framed badly or if it was just difficult. If so, they might toss it out. If nearly everyone gets an item correct, they might toss that out too. As these decisions are undoubtedly made by a committee, so the process is time consuming.

 

[mirimonster]

I'm convinced it's because they want to make sure that the grace period to report a problem with your test has passed. They don't want every pre-med who gets a 20 to immediately call foul in order to have the poor score removed.

 

[AtlasCarraway]

After they see all the test scores they have to "norm" the scores so that they are equivalent to a test taken at another time. Also, if a disproportionate number miss an item, they have to evaluate whether the question was framed badly or if it was just difficult. If so, they might toss it out. If nearly everyone gets an item correct, they might toss that out too. As these decisions are undoubtedly made by a committee, so the process is time consuming.

This.

 

[mvenus929]

For step 1 that might be true, I have not done much research on it, but the MCAT is supposed to be based on a predetermined scale. Even so, with computers doing the work that wouldn't take much time at all...

I don't think missing the same number of questions on different administrations of the MCAT would yield exactly the same score (for instance, missing 5 might yield you a 15 in one section on one test, but maybe a 14 on another test). There are also 'test' questions with every administration that don't count towards your final score.

It should be noted, though, that two people taking Step 1 on the same day may still have completely different tests. They have to be evaluated more than just 'this person missed this number of questions'.

 

[kasho11]

After they see all the test scores they have to "norm" the scores so that they are equivalent to a test taken at another time. Also, if a disproportionate number miss an item, they have to evaluate whether the question was framed badly or if it was just difficult. If so, they might toss it out. If nearly everyone gets an item correct, they might toss that out too. As these decisions are undoubtedly made by a committee, so the process is time consuming.

Are you referring to step 1 or both? For the MCAT someone here recently spoke with an AAMC representative who told them that questions are rated on their difficulty and harder questions are worth more. These points are added together and this raw score is then converted into the 1-15 scale. They do not norm the MCAT, it is scored on a pre-determined "curve" based upon the difficulty of the questions.

I can believe that they might toss out a question for various reasons but if too many people got it wrong they could just rank it as a highly difficult question and vice versa for a question that's too easy. I think I will call them and try and get an answer tomorrow, just bugs me not knowing what they're doing and there are a lot of misconceptions about the "curve".

 

[Praefectus]

After they see all the test scores they have to "norm" the scores so that they are equivalent to a test taken at another time. Also, if a disproportionate number miss an item, they have to evaluate whether the question was framed badly or if it was just difficult. If so, they might toss it out. If nearly everyone gets an item correct, they might toss that out too. As these decisions are undoubtedly made by a committee, so the process is time consuming.

It's called glacial leadership. They teach you about it when you get into a position of authority.

 

[kasho11]

After they see all the test scores they have to "norm" the scores so that they are equivalent to a test taken at another time. Also, if a disproportionate number miss an item, they have to evaluate whether the question was framed badly or if it was just difficult. If so, they might toss it out. If nearly everyone gets an item correct, they might toss that out too. As these decisions are undoubtedly made by a committee, so the process is time consuming.

I am not sure if this is true for Step 1 but I called AAMC and they said this was absolutely not the case. The difficulty is based on a pre-determined scale, more difficult questions are worth more than easier questions. There is no committee that gets together to go over questions to be scored. There are possibly some experimental questions on each test and those questions are analyzed for future use. But for the purposes of your score all questions to be scored are based on this pre-determined scale of difficulty and are then converted into the 1-15 score.

It will be interesting to see how long they take to send your scores after the writing section is removed.

 

[theseeker4]

I am not sure if this is true for Step 1 but I called AAMC and they said this was absolutely not the case. The difficulty is based on a pre-determined scale, more difficult questions are worth more than easier questions. There is no committee that gets together to go over questions to be scored. There are possibly some experimental questions on each test and those questions are analyzed for future use. But for the purposes of your score all questions to be scored are based on this pre-determined scale of difficulty and are then converted into the 1-15 score.

It will be interesting to see how long they take to send your scores after the writing section is removed.

Difficulties are pre-determined, but the actual break-down of the 1-15 scale is determined based on percentile. That conversion is where the normalization occurs. This is not at all at odds with what AAMC told you, since the difficulty can be pre-determined without setting in stone where on the 1-15 scale you would fall based on which questions you get wrong.

 

[kasho11]

Difficulties are pre-determined, but the actual break-down of the 1-15 scale is determined based on percentile. That conversion is where the normalization occurs. This is not at all at odds with what AAMC told you, since the difficulty can be pre-determined without setting in stone where on the 1-15 scale you would fall based on which questions you get wrong.

Are you saying that they normalize the scores people get for each test? Because that is at odds with what they told me. There is no curve or normalization that occurs, they are based on previously determined markers of how people did. 1,000 people could score a 40 on a given test, they don't normalize the testers against each other.

 

[theseeker4]

Are you saying that they normalize the scores people get for each test? Because that is at odds with what they told me. There is no curve or normalization that occurs, they are based on previously determined markers of how people did. 1,000 people could score a 40 on a given test, they don't normalize the testers against each other.

I would like to see the source for that claim, as it is the opposite of everything I have heard before.

 

[mirimonster]

"While there may be small differences in the MCAT exam you took compared to another examinee, the scoring process accounts for these differences so that an 8 earned on physical sciences on one exam means the same thing as an 8 earned on any other exam. The percentile provided on your score report simply indicates what percentage of examinees from the previous testing year scored the same as you did on the MCAT exam.

How you score on the MCAT exam, therefore, is not reflective of the particular exam you took—including the time of day, the test date, or the time of year" - https://www.aamc.org/students/applying/mcat/preparing/85436/preparing_understandingscores.html

They do not normalize the MCAT after the test. The scale is pre-determined by how well the previous testing year did. (That means that they test all questions ahead of time) The scale of your particular test is based on the difficulty of the questions, but that does not mean that each individual question is worth more or less depending on difficulty. If you have more difficult questions on your test, the total scale will reflect that, but still each individual question is worth the same amount of points. Your "raw" score is then converted to your "scaled" score depending on the difficulty of the test. All of this is pre-determined, at least according to their website.

Officially, they take a month to sort out the writing sample. That's why I maintain my previous stance. They don't want to deal with students claiming abnormal testing conditions in order to get out of a poor score.

Edit: This means yes, everyone in your testing center could score a 40 on a given day. The reason they pre-determine the scales is so that if by random chance a bunch of geniuses choose to take the test on the same day, they're not going to kill the curve.

 

[Tots]

I am not sure if this is true for Step 1 but I called AAMC and they said this was absolutely not the case. The difficulty is based on a pre-determined scale, more difficult questions are worth more than easier questions. There is no committee that gets together to go over questions to be scored. There are possibly some experimental questions on each test and those questions are analyzed for future use. But for the purposes of your score all questions to be scored are based on this pre-determined scale of difficulty and are then converted into the 1-15 score.

It will be interesting to see how long they take to send your scores after the writing section is removed.

Pretty sure you misinterpreted the AAMC. Pretty sure questions are all weighted the same, there is no "worth more" for the hard questions. What they meant was they scale for difficulty of exam ie a 48/52 may be a 14 on exam1 and a 13 on exam 2 if exam1 had harder questions.

See this quote from the MCAT essentials:

"Every copy of the testing form measures the same basic skills and concepts. However, every testing form is different in regard to the questions it uses. Because each form has the potential to be slightly more difficult or easy than another, raw scores are converted to a scale that takes into consideration the level of difficulty of test questions in each testing form ensuring equal scoring."

Furthermore just look at the practice MCATs the AAMC puts out. If I remember correctly they claim the scales they give you were pretty much the scales from the real exams - no weighting questions in those scales.

EDIT: mirimonster beat me with a good explanation.

 

[theseeker4]

"While there may be small differences in the MCAT exam you took compared to another examinee, the scoring process accounts for these differences so that an 8 earned on physical sciences on one exam means the same thing as an 8 earned on any other exam. The percentile provided on your score report simply indicates what percentage of examinees from the previous testing year scored the same as you did on the MCAT exam.

How you score on the MCAT exam, therefore, is not reflective of the particular exam you took—including the time of day, the test date, or the time of year" - https://www.aamc.org/students/applying/mcat/preparing/85436/preparing_understandingscores.html

They do not normalize the MCAT after the test. The scale is pre-determined by how well the previous testing year did. (That means that they test all questions ahead of time) The scale of your particular test is based on the difficulty of the questions, but that does not mean that each individual question is worth more or less depending on difficulty. If you have more difficult questions on your test, the total scale will reflect that, but still each individual question is worth the same amount of points. Your "raw" score is then converted to your "scaled" score depending on the difficulty of the test. All of this is pre-determined, at least according to their website.

Officially, they take a month to sort out the writing sample. That's why I maintain my previous stance. They don't want to deal with students claiming abnormal testing conditions in order to get out of a poor score.

Edit: This means yes, everyone in your testing center could score a 40 on a given day. The reason they pre-determine the scales is so that if by random chance a bunch of geniuses choose to take the test on the same day, they're not going to kill the curve.

Interesting, seems my impressions were wrong. I stand corrected