the covid MCAT has the same number of scored questions as the pre-pandemic MCAT

[deleted555445]

FYI, I feel like I've heard people debating this, but AAMC states that the number of graded questions is the same, the only things that have been eliminated are field questions.

See page v of this document, the paragraph on the right side that states, "As you evaluate MCAT scores from the shortened exam this year, it is important to know that the number of scored questions remains the same on the shortened exam."

---

Members don't see this ad.

 

[JimKimSlim]

Yeah, I’m also not sure what to believe. Ask @KnightDoc

 

[KnightDoc]

Yeah, I’m also not sure what to believe. Ask @KnightDoc

Thanks for the tag!!

It's impossible to answer this question with certainty, because AAMC has said two different things. The document you have linked does indeed say that the number of scored questions hasn't changed. AAMC has also published this, which implies that it has in fact changed, without explicitly saying so:

"Students will still be tested on all four sections of the exam and will be responsible for demonstrating the same knowledge and skills at the same levels of difficulty as on the full-length exam. While there is a small reduction in the number of test questions in each test section, the shortened exam maintains the same format and tests the same things as the full-length test. Students will receive the same scores on the shortened exam as they do on the full-length exam. They will receive five scores: one from each of the four sections and one combined total score. Additionally, scores from the shortened exam have the same precision and are reported with the same confidence bands as scores from the full-length exam. Admissions officers will know which students took the shortened exam."

All we know for sure is the test has 38 fewer questions than a full length exam, which does seem like it would be a lot of field test questions. We also know, anecdotally through reddit reports, that people are seeing a much wider variance between their FL average and their actual score (both up and down) than in the past, which would be consistent with fewer questions being scored on the shortened exam.

I don't work for AAMC, so I honestly don't have any greater insight into the issue than this. Also, it doesn't really matter, since there is nothing you can do to prepare differently one way or the other. It is what it is, and there is nothing you can do about it other than not take the test until January, so you really do have to just go with the flow, and realize that it's totally luck of the draw whether the change will help or hurt you. And, of course, if the number of scored questions is exactly the same (which just seems unlikely), the change only helps, since it allows you to complete the test sooner and allows you to avoid being a guinea pig for future examinees with no benefit for you.

 

[deleted555445]

Thanks for the tag!!

It's impossible to answer this question with certainty, because AAMC has said two different things. The document you have linked does indeed say that the number of scored questions hasn't changed. AAMC has also published this, which implies that it has in fact changed, without explicitly saying so:

"Students will still be tested on all four sections of the exam and will be responsible for demonstrating the same knowledge and skills at the same levels of difficulty as on the full-length exam. While there is a small reduction in the number of test questions in each test section, the shortened exam maintains the same format and tests the same things as the full-length test. Students will receive the same scores on the shortened exam as they do on the full-length exam. They will receive five scores: one from each of the four sections and one combined total score. Additionally, scores from the shortened exam have the same precision and are reported with the same confidence bands as scores from the full-length exam. Admissions officers will know which students took the shortened exam."

All we know for sure is the test has 38 fewer questions than a full length exam, which does seem like it would be a lot of field test questions. We also know, anecdotally through reddit reports, that people are seeing a much wider variance between their FL average and their actual score (both up and down) than in the past, which would be consistent with fewer questions being scored on the shortened exam.

I don't work for AAMC, so I honestly don't have any greater insight into the issue than this. Also, it doesn't really matter, since there is nothing you can do to prepare differently one or the other. It is what it is, and there is nothing you can do about it other than not take the test until January, so you really do have to just go with the flow, and realize that it's totally luck of the draw whether the change will help or hurt you. And, of course, if the number of scored questions is exactly the same (which just seems unlikely), the change only helps, since it allows you to complete the test sooner and allows you to avoid being a guinea pig for future examinees with no benefit for you.

Sorry for the confusion, this wasn't a question but more of a statement of something I found. If you check the link in my original post, you'll see a definitive statement that the number of scored questions has not been reduced! Just thought I'd share since I've heard people going back and forth on this.

Edit: I misread your post, I see you were actually responding to my link, not misinterpreting my post as question. I was just sharing what seemed like a definitive answer for people's use.

 

[KnightDoc]

Sorry for the confusion, this wasn't a question but more of a statement of something I found. If you check the link in my original post, you'll see a definitive statement that the number of scored questions has not been reduced! Just thought I'd share since I've heard people going back and forth on this.

Edit: I misread your post, I see you were actually responding to my link, not misinterpreting my post as question. I was just sharing what seemed like a definitive answer for people's use.

Your link is indeed definitive. Would you not agree that my link, which was posted more recently, calls the statement in your link into question?

 

[deleted555445]

Your link is indeed definitive. Would you not agree that my link, which was posted more recently, calls the statement in your link into question?

not really, i agree your quoted section is unclear on its own, but there is an existing answer so i think that takes precedent. especially since they say the precision of the results doesnt change.

 

[KnightDoc]

not really, i agree your quoted section is unclear on its own, but there is an existing answer so i think that takes precedent. especially since they say the precision of the results doesnt change.

Yeah, in the end, it really doesn't matter at all. It just helps explain the recent variances between FLs and the real thing, since less scored questions means each question has a greater impact on the score, both up and down. Just keep in mind they would say that about the reliability of the results in any event. They would never admit this version of the exam was less reliable, even if it was.

Also keep in mind that they do say in the later post that "Admissions officers will know which students took the shortened exam." Why even say this if the shortened exam has the EXACT number of scored questions, and consequently was EXACTLY the same as a full length exam?

Moreover, why not repeat the statement from the earlier post that the number of scored questions didn't change when stating that "... there is a small reduction in the number of test questions in each test section..." if that were true, just to not leave the ambiguity we are debating now? Because they already said it in a different post over two months earlier?

 

[deleted555445]

Yeah, in the end, it really doesn't matter at all. It just helps explain the recent variances between FLs and the real thing, since less scored questions means each question has a greater impact on the score, both up and down.

i agree it doesn't matter, i just thought it would help relieve some anxiety for people who felt nervous a smaller pool of questions might screw them if it focuses on one of their weakness

 

[KnightDoc]

i agree it doesn't matter, i just thought it would help relieve some anxiety for people who felt nervous a smaller pool of questions might screw them if it focuses on one of their weakness

Yup. Regardless of what is true for the actual exam, that is exactly what is happening with the FLs, since we know all of the questions are scored on them and people's scores on the real thing have been varying pretty widely from their FL averages since testing resumed in May.

People just need to realize it's 50-50, because the smaller pool could also help them, as is also being widely reported on reddit. This is why it really doesn't matter, especially because there is nothing to be done, even if there are a smaller number of questions being scored.

 

[deleted555445]

Yup. Regardless of what is true for the actual exam, that is exactly what is happening with the FLs, since we know all of the questions are scored on them and people's scores on the real thing have been varying pretty widely from their FL averages since testing resumed in May.

People just need to realize it's 50-50, because the smaller pool could also help them, as is also being widely reported on reddit. This is why it really doesn't matter, especially because there is nothing to be done, even if there are a smaller number of questions being scored.

i dont think its wise to work backwards from the assumption that people posting on reddit equates a trend, and if you do assume that what is on reddit is representative of an actual larger spread of test scores around an FL average i dont think you can adequately control enough of the variables to say that it is likely explained by a smaller number of test questions. if you controlled for SES maybe you would see the wealthier students are scoring higher than their FL's because they had more time to focus on reviewing them given they had fewer distractions at home leading to more improvement than normal over FL average, while poorer students might feel a lot more test day stress given the state of the economy and the cost of retaking the exam and subsequently do worse than they would from normal exam day pressure. or you could divide people into glasses v non-glasses wearers, and assume many of them did not practice with a mask on, so glasses-wearers do more poorly now that they're distracted by fogging, while non-glasses wearers do better because their scores are being normalized against the glasses-wearers.

it seems i wont change your mind so i wont keep trying to, but for other readers who are wondering i would work under the assumption that there is no difference in the number of scored questions based on statements by AAMC. the biggest difference in the exams in my mind is the lack of a 30 minute break, which is killer.

 

[KnightDoc]

i dont think its wise to work backwards from the assumption that people posting on reddit equates a trend, and if you do assume that what is on reddit is representative of an actual larger spread of test scores around an FL average i dont think you can adequately control enough of the variables to say that it is likely explained by a smaller number of test questions. if you controlled for SES maybe you would see the wealthier students are scoring higher than their FL's because they had more time to focus on reviewing them given they had fewer distractions at home leading to more improvement than normal over FL average, while poorer students might feel a lot more test day stress given the state of the economy and the cost of retaking the exam and subsequently do worse than they would from normal exam day pressure. or you could divide people into glasses v non-glasses wearers, and assume many of them did not practice with a mask on, so glasses-wearers do more poorly now that they're distracted by fogging, while non-glasses wearers do better because their scores are being normalized against the glasses-wearers.

it seems i wont change your mind so i wont keep trying to, but for other readers who are wondering i would work under the assumption that there is no difference in the number of scored questions based on statements by AAMC. the biggest difference in the exams in my mind is the lack of a 30 minute break, which is killer.

I'm not a huge fan of reddit, but it does have a lot of participation, so it does present a decent sample size. Of course, it is far from random.

My point is simply that it has a rich history of creating threads with reactions from each and every administration of the test. Actual reported scores were much closer to FL averages prior to the shortened test. Again, regardless of whether or not the shortened test has the same number of scored questions as the full length exam, it definitely has less scored questions than the FLs, and the reported actual scores are now varying a lot more from FL averages than before.

Yes, we don't know if this is true for each and every test taker, but we do know it's true for a decent sample size reporting on reddit. The variances have been reported both up and down, so I don't see the need to normalize for SES, fogging, or anything else. I'll assume if low SES are having issues preparing for the exam, that would also be reflected in low FL averages, and it wouldn't explain a large downward test day variance from that low FL average. Just pulling theories out of thin air to potentially explain any deviation just doesn't seem productive. Maybe high SES went into the test overconfident and scored lower than expected? Maybe low SES had such low FL averages that their performance has nowhere to go but up? Any of us can come up with a theory for anything!!

Variances are a fact of life, but they are greater now, and each section now has 48 questions whereas the FLs had 59 questions (53 for CARS). My conclusion is that the 38 less questions as compared to the FLs is causing the impact of incorrect answers to be greater than before. If you were scoring around 80% on the FLs and if you would have missed 4 of the 11 questions that aren't now asked, your score will now spike up in that section, and vice versa.

Of course, if AAMC isn't BSing us, the 38 missing questions were never scored, so it shouldn't make a difference, and yet, differences are being reported. I honestly don't think AAMC would admit to a problem if there was one, because that would destroy confidence in the exam. Only they know if there were really 38 field test questions on the full length exam, not 15, or 25, or 35.

In May they said the number of scored questions would remain the same. They removed that language from their July 23rd statement. I'm taking that change, plus the reported variances, to mean they are scoring less questions while swearing the scores are indistinguishable from full length scores, and also stating that schools will know who took a shortened exam. Of course they will, simply by looking at the date! But why even say that if it's exactly the same test as before?

So is it really the same test? What's the difference? It's the test we now have, and, as I said before, if there is a difference, it is as likely to help as hurt an individual candidate.

 

[deleted555445]

I'm not a huge fan of reddit, but it does have a lot of participation, so it does present a decent sample size. Of course, it is far from random.

My point is simply that it has a rich history of creating threads with reactions from each and every administration of the test. Actual reported scores were much closer to FL averages prior to the shortened test. Again, regardless of whether or not the shortened test has the same number of scored questions as the full length exam, it definitely has less scored questions than the FLs, and the reported actual scores are now varying a lot more from FL averages than before.

Yes, we don't know if this is true for each and every test taker, but we do know it's true for a decent sample size reporting on reddit. The variances have been reported both up and down, so I don't see the need to normalize for SES, fogging, or anything else. I'll assume if low SES are having issues preparing for the exam, that would also be reflected in low FL averages, and it wouldn't explain a large downward test day variance from that low FL average. Just pulling theories out of thin air to potentially explain any deviation just doesn't seem productive. Maybe high SES went into the test overconfident and scored lower than expected? Maybe low SES had such low FL averages that their performance has nowhere to go but up? Any of us can come up with a theory for anything!!

Variances are a fact of life, but they are greater now, and each section now has 48 questions whereas the FLs had 59 questions (53 for CARS). My conclusion is that the 38 less questions as compared to the FLs is causing the impact of incorrect answers to be greater than before. If you were scoring around 80% on the FLs and if you would have missed 4 of the 11 questions that aren't now asked, your score will now spike up in that section, and vice versa.

Of course, if AAMC isn't BSing us, the 38 missing questions were never scored, so it shouldn't make a difference, and yet, differences are being reported. I honestly don't think AAMC would admit to a problem if there was one, because that would destroy confidence in the exam. Only they know if there were really 38 field test questions on the full length exam, not 15, or 25, or 35.

In May they said the number of scored questions would remain the same. They removed that language from their July 23rd statement. I'm taking that change, plus the reported variances, to mean they are scoring less questions while swearing the scores are indistinguishable from full length scores, and also stating that schools will know who took a shortened exam. Of course they will, simply by looking at the date! But why even say that if it's exactly the same test as before?

So is it really the same test? What's the difference? It's the test we now have, and, as I said before, if there is a difference, it is as likely to help as hurt an individual candidate.

have you done any analysis to say that the variance reported on reddit is higher this year? have you thought about whether there might be some reporting bias on reddit? maybe someone scored lower than they expected, went on reddit to complain about how it was the shorter exams fault, and now everyone who was 3 points +/- their FL score wants to share their story, leading to a whole thread of people saying they are higher or lower than their FL average who in past years wouldn't have stopped to share their difference.

the bolded statements above are exactly that point i was making. if you assume there to be larger variance from FL scores, which i don't agree is true in the slightest, explaining that by saying its due to fewer scored questions has the same validity as any theory that can't be proven. i could say that AAMC decided to give each test a "flavor" where they focus 30% on one specific subject, and someone who reviewed the pentose phosphate pathway heavily does better than they would have on a normal exam and someone who only skimmed it doesn't do as well as they would have if the subject matter had been more equal. AAMC hasn't said they did this, but it explains the finding.

to answer your question, AAMC said in your link that schools would know who took the shorter exam so that, "they understand that some of their applicants have had disrupted learning and experienced multiple disruptions in preparing for the MCAT exam." obviously they would know by seeing the data 2020 on there, it was just some classic needless writing from a bureaucratic monolith.

long story short, if AAMC comes out explicitly saying there are fewer scored questions, i will agree with your point. but until then, i'll defer to them explicitly saying there will not be fewer scored questions. i think we've reached the end of usefulness of this thread, so i'll give you a chance to respond if you would like then i will close it

 

[KnightDoc]

have you done any analysis to say that the variance reported on reddit is higher this year? have you thought about whether there might be some reporting bias on reddit? maybe someone scored lower than they expected, went on reddit to complain about how it was the shorter exams fault, and now everyone who was 3 points +/- their FL score wants to share their story, leading to a whole thread of people saying they are higher or lower than their FL average who in past years wouldn't have stopped to share their difference.

the bolded statements above are exactly that point i was making. if you assume there to be larger variance from FL scores, which i don't agree is true in the slightest, explaining that by saying its due to fewer scored questions has the same validity as any theory that can't be proven. i could say that AAMC decided to give each test a "flavor" where they focus 30% on one specific subject, and someone who reviewed the pentose phosphate pathway heavily does better than they would have on a normal exam and someone who only skimmed it doesn't do as well as they would have if the subject matter had been more equal. AAMC hasn't said they did this, but it explains the finding.

to answer your question, AAMC said in your link that schools would know who took the shorter exam so that, "they understand that some of their applicants have had disrupted learning and experienced multiple disruptions in preparing for the MCAT exam." obviously they would know by seeing the data 2020 on there, it was just some classic needless writing from a bureaucratic monolith.

long story short, if AAMC comes out explicitly saying there are fewer scored questions, i will agree with your point. but until then, i'll defer to them explicitly saying there will not be fewer scored questions. i think we've reached the end of usefulness of this thread, so i'll give you a chance to respond if you would like then i will close it

All fair points. No, I have not done a statistical analysis, and no, people by and large are not complaining on reddit about the shortened exam. It's just that a lot of people are expressing surprise, both up and down, about how far their actual scores are from their FL averages.

The fewer scored questions theory is all mine, and, as I've said repeatedly, it works both ways, so it's neither good nor bad. It's just something to be aware of. If there are, in fact, fewer scored questions, it would absolutely explain why a large cross section of people are scoring way beyond the confidence band, both up and down, on just another AAMC FL.

Remember, the FLs are retired actual exams. So, while any given person's performance might improve over time, the average of four of these tests taken in close proximity to each other should map pretty closely to the fifth test taken during the same period, assuming the confidence bands are valid. Sure there is extra pressure taking a test that counts in a test center as compared to practice tests at home, but that just doesn't explain why so many people are now +/- 5, 6, 7 or more points away from their FL averages now when a lot of people were +/- 0, 1, 2 points away before May.

 

[deleted1044860]

I scored a 132 in CARS on every single aamc practice exam (taking all of them under testing conditions), including the one I took as a "diagnostic" before beginning any mcat studying. I got a 130 on the actual shortened MCAT exam. This is just an anecdote of course, but upon seeing this score, my gut told me that this shortened mcat must have had fewer scored questions than a typical exam to account for the unexpected result.

No complaints here though. I'm so grateful that I finally got to take the exam (and now apply to med school) after cancellations! I'm thankful to aamc for figuring out a way to make it so that students could still take the exam during this pandemic (although reports of students testing positive for covid after taking the mcat are very concerning).

I'm sharing my experience here because I do think it is helpful for future test takers this cycle to be aware that what they're taking might be different than what they are expecting.

 

[KnightDoc]

I scored a 132 in CARS on every single aamc practice exam (taking all of them under testing conditions), including the one I took as a "diagnostic" before beginning any mcat studying. I got a 130 on the actual shortened MCAT exam. This is just an anecdote of course, but upon seeing this score, my gut told me that this shortened mcat must have had fewer scored questions than a typical exam to account for the unexpected result.

No complaints here though. I'm so grateful that I finally got to take the exam (and now apply to med school) after cancellations! I'm thankful to aamc for figuring out a way to make it so that students could still take the exam during this pandemic (although reports of students testing positive for covid after taking the mcat are very concerning).

I'm sharing my experience here because I do think it is helpful for future test takers this cycle to be aware that what they're taking might be different than what they are expecting.

And how far off was your total from your FL average? If it was more than 2-3 points, it was beyond the confidence band, and you are another data point supporting my theory.

 

[deleted1044860]

No, my total mcat score wasn't more than 2-3 points from my FL average. My total mcat score was within range of my practice exams. My first two FL scores were significantly lower than the second two scores because I studied a lot in between them, so I'm not sure my average scores are a helpful data point.

 

[KnightDoc]

No, my total mcat score wasn't more than 2-3 points from my FL average. My total mcat score was within range of my practice exams. My first two FL scores were significantly lower than the second two scores because I studied a lot in between them, so I'm not sure my average scores are a helpful data point.

Yeah, then you're fine. A 2 point deviation on one section is not evidence of less scored questions. Sampling variations could easily account for that. Especially with a 132, you had nowhere to go but down. One passage giving you a problem on the real thing is all it would take, and that would have nothing to do with how many questions were scored.

Also, most people see some improvement as they practice and get better. That's why we take an average, and don't just rely on our final, usually highest score. The highest score can give false confidence while the average has been pretty accurate at setting realistic expectations, at least until recently. In your case, you scored right within expectations, notwithstanding your CARS disappointment.

 

[deleted1044860]

That's a very good point!