Is the MCAT Poorly Constructed?

[DB12]

The AAMC's attempt to assess the "reasoning capacity" a test-taker has, runs the risk of asking increasingly flawed or ambiguous questions and adopting completely arbitrary answers. The most salient tenet to any scientific inquiry asserts the fact that not everything is completely understood. The earliest Greek philosophers (who developed the scientific method) assumed everything is unknown, until provided concrete evidence and developed hypothesizes strictly from the evidence provided. If we lack needed evidence to develop any hypothesis, we must proclaim such is not known. I fear the AAMC is abandoning this most salient tenet in the development of the MCAT and provide two examples to this end.

The CARS section is the most poorly constructed section in the MCAT. Providing a 4-5 paragraph excerpt of a larger body of abstract text and requiring a test-taker to apprehend what arbitrary answers the AAMC adopted, is akin to providing a 1inchx1inch piece of a 10inchx10inch Jackson Pollock painting and asking what the artist attempted to express in the painting. A well constructed test question must adopt a correct answer that is capable of being cross checked by the test writer. If you write question about the definition of a diastereomer, you must be capable of cross checking the answer you adopt as correct, with an authoritative source (like the IUPAC Gold Book). If you write a question about a laboratory technique, assertions you adopt as accurate must be cross checked with claims made in US Patent Applications for said techniques. The CARS section is not constructed in a manner which the AAMC may cross check an answer it adopts as correct, which makes its construction purely arbitrary.

Pre-Med, Pre-Vet and Pre-DPM students should confront the AAMC about how the MCAT is being curated, peer-reviewed and subsequently marketed. Keep in mind, the AAMC has a commercial relationship with the Test-Taker in the administration of the MCAT, Not the allopathic/osteopathic, veterinary or podiatric medical schools.

***Update***
Along with others, I have submitted a formal complaint to the US Federal Trade Commission pertaining to the AAMC's mismarketing of its MCAT product and in-ability to do what it claims about the MCAT. Feel free to see a sample of part of the complaint below (the preceding pages were omitted as they contain evidence being used against the AAMC).

 

[Princeton Medical Student]

No. I have yet to encounter even a single answer (even on CARS) that after review has an objectively better or equally valid answer. None of the answers I have seen on all of the AAMC prep material, nor on the actual test required knowledge either outside of the passage, or knowledge that isn't on the content outline.

 

[Matthew9Thirtyfive]

I would argue the CARS section is the greatest weakness the MCAT has. The AAMC will take a 4-5 paragraph excerpt from a larger body of very abstract texts and have you "Reason Beyond the Text" by "Applying or Extrapolating Ideas From the Passage to New Contexts" and "Assessing the Impact of Incorporating New Factors, Information, or Conditions on Ideas From the Passage." This is akin to giving someone a 1inchx1inch piece of a 10inchx10inch Jackson Pollock painting and asking them what the artist was attempting to express in his painting (an entirely arbitrary process).

A far more reasoned test that does not engage in the arbitrary pitfalls of the MCAT would be the LSAT, it is based on "If-Then statements" and requires the execution of logical reasoning absent any arbitrary applications.

The LSAT also has CARS like sections. The logic games are fun, but the problem is they can totally be trained on. There are a finite number of different games and memorizing the strategies for them essentially means you just have to recognize the game type and apply the strategy.

 

[DB12]

An appropriately developed test should be capable of assessing scholastic conditioning, including memorization of strategies to answer questions. Moreover, the test-writer must be capable of cross checking the answer it asserts is correct, with an outside source.

 

[KnightDoc]

An appropriately developed test should be capable of assessing scholastic conditioning, including memorization of strategies to answer questions with definitive answers

As a fellow test taker, I think the flaw in your reasoning is your assumption that the test is designed to either be a reasoning test OR a test of scientific knowledge. I don't believe that is the case. I am pretty sure the purpose of the test is to stratify the applicant pool in such a way as to predict success in the pre-clinical portion of medical school.

By this measure, the test does precisely what it was designed to do. Multiple studies have shown a high correlation to Step 1 scores, which are supposed to measure pre-clinical academic progress, and the distribution of scores, far from being random, is highly consistent, test date to test date, and year to year. This undermines your argument that there are no objectively correct answers to the test, since the distribution of scores would be unpredictable and random if this were the case. The test takes around 70,000 test takers per year and distributes them on a bell curve to give schools a consistent metric beyond grades to predict, with a reasonable degree of accuracy, how people will perform in the pre-clinical years of med school.

That is all it is designed to do. The fact that it seems like gibberish to some test takers is exactly the point. If it were nothing more than an objective knowledge test, it would be much more difficult to achieve the desired score distribution (given how smart a significant portion of the pool is), and would lose its value as a key, uniform tool for adcoms in making decisions. It also wouldn't map nearly as well to pre-clinical medical education.

 

[DB12]

As a fellow test taker, I think the flaw in your reasoning is your assumption that the test is designed to either be a reasoning test OR a test of scientific knowledge. I don't believe that is the case. I am pretty sure the purpose of the test is to stratify the applicant pool in such a way as to predict success in the pre-clinical portion of medical school.

By this measure, the test does precisely what it was designed to do. Multiple studies have shown a high correlation to Step 1 scores, which are supposed to measure pre-clinical academic progress, and the distribution of scores, far from being random, is highly consistent, test date to test date, and year to year. This undermines your argument that there are no objectively correct answers to the test, since the distribution of scores would be unpredictable and random if this were the case. The test takes around 70,000 test takers per year and distributes them on a bell curve to give schools a consistent metric beyond grades to predict, with a reasonable degree of accuracy, how people will perform in the pre-clinical years of med school.

That is all it is designed to do. The fact that it seems like gibberish to some test takers is exactly the point. If it were nothing more than an objective knowledge test, it would be much more difficult to achieve the desired score distribution (given how smart a significant portion of the pool is), and would lose its value as a key, uniform tool for adcoms in making decisions. It also wouldn't map nearly as well to clinical medical education.

My apologies, I ended up editing the original post.

How does the AAMC cross check the answers it adopts as accurate in the CARS section?

 

[JimKimSlim]

I don’t think MCAT successfully predicts ones capacity to become a good doctor, but it IS a great equalizer among the applicant pool. My school is known for heavily deflating grades, and ad coms know that but don’t really treat your app differently from other applicants.

I have seen some AAMC explanations that I disagreed with, but only few test takers would get those problems right and didn’t really end up mattering that much in terms of final grade. It’s not just you who feels that way when it comes to CARS. In fact, i believe CARS tests your ability to read test makers mind rather than critical thinking... Here’s an N=1, but I aced SAT reading section without prepping at all in high school, but I got 122 on CARS on my diagnostic test without any prep...

 

[DB12]

I don’t think MCAT successfully predicts ones capacity to become a good doctor, but it IS a great equalizer among the applicant pool. My school is known for heavily deflating grades, and ad coms know that but don’t really treat your app differently from other applicants.

I have seen some AAMC explanations that I disagreed with, but only few test takers would get those problems right and didn’t really end up mattering that much in terms of final grade. It’s not just you who feels that way when it comes to CARS. In fact, i believe CARS tests your ability to read test makers mind rather than critical thinking... Here’s an N=1, but I aced SAT reading section without prepping at all in high school, but I got 122 on CARS on my diagnostic test without any prep...

I agree, but the AAMC's departures from sound scientific reasoning and intentional construction of an entire section of the test which is completely arbitrary is absurd. I have no clue how the AAMC has not been called-out on this.

 

[JimKimSlim]

I agree, but the AAMC's departures from sound scientific reasoning and intentional construction of an entire section of the test which is completely arbitrary is absurd. I have no clue how the AAMC has not been called-out on this.

Get used to AAMC getting away with absurd changes/ideas, as long as you're premed!

For example. AAMC added VITA this admissions cycle, which is just you answering questions while smiling at a computer screen (I believe it's just another hurdle to stress pre-med students out).

 

[DB12]

I do not believe it should be tolerated by pre-Meds.

It is inappropriate for the AAMC to market and sell a product (the MCAT) it knows is flawed.

 

[KnightDoc]

My apologies, I ended up editing the original post.

How does the AAMC cross check the answers it adopts as accurate in the CARS section?

It writes the questions and the answers -- that's how!!! There is nothing to cross check. It's not like they are getting it from a third party and taking it themselves and randomly deciding which are the correct answers, although I see how you might think it looks like that!

I think it's counterproductive to bang your head against the wall on this. It needs to be very difficult, given how many very smart people take it and the need to have a wide distribution of scores, with a thin tail at the upper end. You don't have to ace the test to do well -- you just have to do better than half of the other test takers. That will get you a 125 in CARS, which is considered acceptable by most schools.

Otherwise, while the test is undoubtedly difficult, and CARS is considered by many to be the most difficult section, at least half of all other test takers are seeing something you are not. This is not an accident. This is by design in order to achieve the desired score distribution.

 

[DB12]

It writes the questions and the answers -- that's how!!! There is nothing to cross check. It's not like they are getting it from a third party and taking it themselves and randomly deciding which are the correct answers, although I see how you might think it looks like that!

I think it's counterproductive to bang your head against the wall on this. It needs to be very difficult, given how many very smart people take it and the need to have a wide distribution of scores, with a thin tail at the upper end. You don't have to ace the test to do well -- you just have to do better than half of the other test takers. That will get you a 125 in CARS, which is considered acceptable by most schools.

Otherwise, while the test is undoubtedly difficult, and CARS is considered by many to be the most difficult section, at least half of all other test takers are seeing something you are not. This is not an accident. This is by design in order to achieve the desired score distribution

A test does not have to be arbitrary to be difficult. The concern about developing a test question where the asserted correct answer can not be cross checked, is that it is impossible for the test writer to defend why they chose a particular answer as correct.

In science, academia, medicine etc, one must be capable of citing sources, cross checking answers and asserting facts that have been proven. The AAMC is capable of doing none of this in the development of the CARS section.

Your argument that the test must be difficult does not excuse the AAMC from developing an inordinately arbitrary test.

 

[KnightDoc]

A test does not have to be arbitrary to be difficult. The concern about developing a test question where the asserted correct answer can not be cross checked, is that it is impossible for the test writer to defend why they chose a particular answer as correct.

In science, academia, medicine etc, one must be capable of citing sources, cross checking answers and asserting facts that have been proven. The AAMC is capable of doing none of this in the development of the CARS section.

Your argument that the test must be difficult does not excuse the AAMC from developing an inordinately arbitrary test.

This is very true!!!! My argument is that if the test were truly arbitrary, it would not have the same consistent bell distribution of scores from form to form, date to date, and year to year, because there would be no way for anyone, let alone a very consistent slice of test takers, to do well by randomly correctly guessing the random answers.

If it were truly arbitrary, we'd all be scoring around 25%, with a small, random number of us deviating from that, up and down, on each form of the exam, with no pattern other than a random one. The fact that a small slice of people consistently do well, with a comparable small slice doing poorly, and the vast majority of test takers clustered in the middle, indicates that it is not "inordinately arbitrary," but rather is simply difficult by design.

 

[DB12]

I must re-itterate the fact the AAMC is not capable of defending the answers it adopts as correct in the CARS section as it remains the sole arbiter as to what it feels is the most right or wrong. The AAMC is requesting the test-taker apprehend the opinions of the test-writer.

You can make a bell curve as long as there are at least three (3) different scores among a test sample of three people or three million. If only three people take the CARS and score 24%, 25% and 26% correct respectively,
the one who answered 26% correct would get 132,
the one who answered 25% correct would get 125
the one who answered 24% correct would get 118

 

[DB12]

No. I have yet to encounter even a single answer (even on CARS) that after review has an objectively better or equally valid answer. None of the answers I have seen on all of the AAMC prep material, nor on the actual test required knowledge either outside of the passage, or knowledge that isn't on the content outline.

Princeton Medical Student,

You know Princeton University has no medical school, correct?

 

[lanzhou_lamian]

I thought it was fine, CARS included. Between AAMC and third party study materials, I was able to better understand what CARS questions were looking for, and I generally found the answer explanations satisfactory.

In regards to the other three sections, 95% of questions aligned with the recommended content review. It look some practice to get my passage reading speed up, but that was the biggest hurdle. Again, I think reading the answer explanations to understand both why the correct answer is right and why the others are not will hopefully make the test feel less arbitrary to you.

 

[deleted1085158]

I think the MCAT can be prepared for but my complaint with it is that it is a test that clearly favors those who have money. If you spend upwards of $10000, you can buy hundreds of hours of 1 on 1 teaching that can help you with preparing for the exam the right way as opposed to someone who can only afford to watch Khan Academy videos for free.

In that sense, I would say it is not a fair exam but life is also not fair.

 

[Princeton Medical Student]

Princeton Medical Student,

You know Princeton University has no medical school, correct?

Of course it does. Across the street from the law campus, affiliated with Princeton Plainsboro.

 

[Princeton Medical Student]

I must re-itterate the fact the AAMC is not capable of defending the answers it adopts as correct in the CARS section as it remains the sole arbitor as to what it feels is the most right or wrong. The AAMC is requesting the test-taker apprehend the opinions of the test-writer. You can make a bell curve as long as there are at least three different scores among a test sample of three people or three million.

Because the test is curved, the majority of people may do very poorly but the lucky few that apprehended whatever arbitrary answer the AAMC adopts will be given the higher curved score (which is the only score made available to the test-taker).

Honestly you are overthinking this. There are very few answers that have two decent answers, and even those that do have a better answer. I have yet to encounter even one CARS question where I could contend that my answer was more correct. I welcome you to change my mind on any of these from any official CARS prep material.

 

[JimKimSlim]

Honestly you are overthinking this. There are very few answers that have two decent answers, and even those that do have a better answer. I have yet to encounter even one CARS question where I could contend that my answer was more correct. I welcome you to change my mind on any of these from any official CARS prep material.

I mean we have different brains, so what seems ambiguous and wrong to me might seem clear like a keratin pearl to you. I don't like CARS, but it's just a test written by another human being. Once you get the hang of the test maker's thought-process and rationale, it becomes tolerable. I don't think that's a good measure of critical thinking ability, but again we have different brains.

 

[DB12]

Honestly you are overthinking this. There are very few answers that have two decent answers, and even those that do have a better answer. I have yet to encounter even one CARS question where I could contend that my answer was more correct. I welcome you to change my mind on any of these from any official CARS prep material.

How does the AAMC cross check an answer it asserts as correct in CARS?

I am forbidden from disclosing exact prep material from the AAMC, but imagine if they asked you to assume:
What early native Americans would have made of opera after reading an excerpt about a writer's critique modern opera is too commercialized?
A) Native Americans would like modern opera
B) Native Americans would like contemporary opera
C) Native Americans would not like any opera
D) Native Americans would like any opera

Here are a few issues with the question asked and each of the answers provided:
1) How does the test-writer define "native Americans?"
2) How does the test-writer define "modern opera?"
3) Does "contemporary" mean our modern day opera shared by the test-taker or that of a specific time shared by native American's the test-writer did not disclose?
4) To what extent are we assuming "native Americans" like or dislike "commercialization?"

Now in the instance of the above example, the test writer adopted answer option C) as the correct answer. The test writer made a broad assumption (which the test writer expected you to apprehend) native Americans were akin to a modern-day culture, resistant to assimilating with new cultures, would find the opera foreign and therefore universally dislike any opera.

The test-writer has adopted answer option C) as the correct answer and attempts to cross check the answer. When the test-writer is not capable of finding a poll of early native Americans' opinions on opera over the last two hundred years, the test-writer tells you to pound sand and claims their test is a "reasoning" test. The extent you scrutinize each answer option will dictate which one you choose. All of this will be futile unless you scrutinize only to the extent of the test-writer, no further and no less. This is Not an appropriate way to construct a standardized test!

 

[DB12]

I thought it was fine, CARS included. Between AAMC and third party study materials, I was able to better understand what CARS questions were looking for, and I generally found the answer explanations satisfactory.

In regards to the other three sections, 95% of questions aligned with the recommended content review. It look some practice to get my passage reading speed up, but that was the biggest hurdle. Again, I think reading the answer explanations to understand both why the correct answer is right and why the others are not will hopefully make the test feel less arbitrary to you.

I would argue the biology, chemistry and physics questions are typically well written. The AAMC does make a few assertions that vary or ignore official IUPAC definitions, USP monograph data, FDA drug label data or claims made in US Patent Applications about laboratory techniques that must be addressed (keep in mind, the aforementioned sources are the ultimate authority as to what may be claimed about chemicals, laboratory techniques, physiology ect...).

I stand by my admonishments of the CARS section, it is poorly constructed from the approach, what it attempts to assess about a test-taker and its in-ability to be peer-reviewed/cross checked.

 

[lanzhou_lamian]

I would argue the biology, chemistry and physics questions are typically well written. The AAMC does make a few assertions that vary or ignore official IUPAC definitions, USP monograph data, FDA drug label data or claims made in US Patent Applications about laboratory techniques that must be addressed (keep in mind, the aforementioned sources are the ultimate authority as to what may be claimed about chemicals, laboratory techniques, physiology ect...).

I stand by my admonishments of the CARS section, it is poorly constructed from the approach, what it attempts to assess about a test-taker and its in-ability to be peer-reviewed/cross checked.

To each their own I guess? I had no issue with the way CARS was constructed and didn't find it overly ambiguous. But best of luck studying for the exam.

 

[KnightDoc]

The test-writer has adopted answer option C) as the correct answer and attempts to cross check the answer. When the test-writer is not capable of finding a poll of early native Americans' opinions on opera over the last two hundred years, the test-writer tells you to pound sand and claims their test is a "reasoning" test. The extent you scrutinize each answer option will dictate which one you choose. All of this will be futile unless you scrutinize only to the extent of the test-writer, no further and no less. This is Not an appropriate way to construct a standardized test!

BS! This is EXACTLY the appropriate way to construct a standardized test, and your failure to grasp this probably explains the disconnect between what you expect from the exam and reality.

AAMC provides an outline of science topics that will be tested, which requires lots of knowledge of concrete information, which is why you are okay with those sections. For CARS, you are not expected to be aware of outside information on an unlimited variety of topics, such as opinion polls on early (or modern) Native American attitudes towards anything. By the way, for the record, I'm sure there were no opinion polls taken of early Native Americans regarding anything, so the external validation you desire is impossible.

The whole idea is to test your critical analysis and reasoning skills (CARS = Critical Analysis and Reasoning Skills!!!!! ) by providing you information and testing your ability to synthesize the information provided and reason through the correct answers, using nothing other than your brain and the information provided, through CRITICAL ANALYSIS AND REASONING, not reference to outside information. YES!!!! The test is measuring your ability to "scrutinize only to the extent of the test-writer, no further and no less." That's what creates a level playing field, without regard to whether you are a Native American expert or don't even know what one is.

If you can't do it naturally, it's a skill you need to develop by practice. Seeking validation beyond the test passage is where you are making your mistake. Think of it like this -- it's fiction (even if it isn't!), and the entire universe of information necessary to answer the question is provided in the passage. Actual Native American attitudes are irrelevant, so cross referencing external data won't get you where you need to be. The only relevant information is provided in the passage, and legitimate inferences that can be drawn therefrom. Insisting on more will only lead to failure and disappointment.

 

[DB12]

BS! This is EXACTLY the appropriate way to construct a standardized test, and your failure to grasp this probably explains the disconnect between what you expect from the exam and reality.

AAMC provides an outline of science topics that will be tested, which requires lots of knowledge of concrete information, which is why you are okay with those sections. For CARS, you are not expected to be aware of outside information on an unlimited variety of topics, such as opinion polls on early (or modern) Native American attitudes towards anything. By the way, for the record, I'm sure there were no opinion polls taken of early Native Americans regarding anything, so the external validation you desire is impossible.

The whole idea is to test your critical analysis and reasoning skills (CARS = Critical Analysis and Reasoning Skills!!!!! ) by providing you information and testing your ability to synthesize the information provided and reason through the correct answers, using nothing other than your brain and the information provided, through CRITICAL ANALYSIS AND REASONING, not reference to outside information. YES!!!! The test is measuring your ability to "scrutinize only to the extent of the test-writer, no further and no less." That's what creates a level playing field, without regard to whether you are a Native American expert or don't even know what one is.

If you can't do it naturally, it's a skill you need to develop by practice. Seeking validation beyond the test passage is where you are making your mistake. Think of it like this -- it's fiction (even if it isn't!), and the entire universe of information necessary to answer the question is provided in the passage. Actual Native American attitudes are irrelevant, so cross referencing external data won't get you where you need to be. The only relevant information is provided in the passage, and legitimate inferences that can be drawn therefrom. Insisting on more will only lead to failure and disappointment.

I am not saying you need to be aware of outside information to answer the question accurately, but the words used in the question and answer must be defined in either the passage or the question/answer. If the test-writer does not define what they mean by "native Americans," timeframes shared in "contemporary" opera, inferences to native Americans' opinions on commercialization, it is impossible to form a well-reasoned assumption of such,

Appropriate Critical Analysis and Reasoning Skills must be capable of determining which test takers are capable of reserving premature assumptions on a subject matter that has not been completely developed. Although the AAMC feels assumptions can be made in any instance, this is an opinion they may have but not shared by everyone and the adoption of their opinion into the construct of the CARS in the MCAT is purely arbitrary.

It is dangerous in medicine to feel an assumption can be made in any instance. According to the scientific method, it is more important to know when an assumption can not be made.

 

[vext840]

I know this is off topic, but... which of the earliest Greek philosophers developed the scientific method? Was it Thales who thought that everything was made of water? Or Aristotle (I know, I know, not that early) who thought that everything could be deduced from assumed first principles? Or was it the earliest of the philosophers, Sir Francis Bacon?

 

[neurobiogirl2022]

Of course it does. Across the street from the law campus, affiliated with Princeton Plainsboro.

Yay a fellow House M.D. fan

 

[DB12]

I know this is off topic, but... which of the earliest Greek philosophers developed the scientific method? Was it Thales who thought that everything was made of water? Or Aristotle (I know, I know, not that early) who thought that everything could be deduced from assumed first principles? Or was it the earliest of the philosophers, Sir Francis Bacon?

Plato, in his socratic paradox.

 

[vext840]

Though it's wonderful to celebrate the Greeks for pursuing knowledge, truth, and beauty in ways that continue to inspire, the structure, process, and goals of Socratic dialogue seem pretty different to me than the scientific method we employ today. Aporia is powerful and humbling, but it's not, as we now use term, scientific.

 

[DB12]

Though it's wonderful to celebrate the Greeks for pursuing knowledge, truth, and beauty in ways that continue to inspire, the structure, process, and goals of Socratic dialogue seem pretty different to me than the scientific method we employ today. Aporia is powerful and humbling, but it's not, as we now use term, scientific.

From whom do you think others developed the scientific method?

Knowing what exactly it is you do not know is paramount in any reasoning process, which what the AAMC categorically misapprehends in its construction of the CARS section in the MCAT.

The Plato/Socratic doctrine is very applicable in modern approaches to medicine and other sciences. Many clinical pitfalls arise from prematurely assuming things or refusing to accept the fact a matter is not known. I encourage everyone to read FDA label data about drug products, medical devices or testing. You would be surprised to learn what little a manufacturer is allowed to claim about a product. This is because in science we do not delve in the realms of any assumptions.