USMLE versus MCAT difficulty?

[AnatomyGrey12]

So really what I know is that among my friends and peers and the people training me, suggesting a 255 will do better for their patients than a 240 is absolutely laughable.

Yeah that's not really what the argument is though, and there is definitely a difference between a 255 and say a 230. I agree USMLE is not good at separating out the top end of the bell curve, but honestly I don't really think that matters as much as people think it does. You also happen to be surrounded by mostly people at the right end of the bell curve so I think your viewpoint is skewed by where you are training.

Yet, that has a huge impact on matching competitive specialties. Add into this that they could both be true 248s and one got a lot luckier, which happens 1/3rd of the time, and it looks to me like a system that absolutely cannot be allowed to continue in it's current form.

But does it though? You go to Hopkins if I'm not mistaken. Do the people with 240s not match competitive specialties and the people with 250s do? I doubt it. This often gets thrown around but the charting outcomes and match data say otherwise. Step scores are important no doubt but not getting a 250 isn't a death sentence for ANY specialty. I think the importance is overstated due to medical student neuroticism. People with 240s, and even 230s, match competitive specialties at high rates. If you have a 240 and the rest of your application is in order then you have an excellent chance of matching X specialty of your choice.

What is your metric for "better" on wards? If it's regurgitation of factoids to answer pimp questions then I would imagine there is a heavy correlation btw step scores and performance in this area. I'm vehemently against step going p/f for most of the reasons you've identified; it screws over students from lower tier schools (along with students who chose their state school for financial reasons). With that said, much like the mcat I don't think Step1 meaningfully identifies who will be a good doctor vs. not - there's just too many subjective qualities that factor in - but there's no way to rely solely on these qualities without leaving the door open to nepotism etc.

I mean better on the wards as in they give better and more concise presentations to the attendings, they can get assigned a new patient and take a quality history and do a solid exam, they are constantly creating realistic and high quality differentials on their patients. Essentially they do the exact same things as any other student but they do so much more effectively than everyone else. You know those students that get told by attendings, "I'm impressed with how you think, you already approach patients like an intern should" midway through 3rd year? Those people almost always are people who did very well on boards.

As to the bolded I don't think anyone is really arguing that. Standardized tests are always just a mediocre way of assessing stuff like that, but USMLE is actually roughly correlated to board pass rates and the content is honestly pretty dang relevant to medicine. So when you have large swaths of people applying to residency programs that all look the exact same you need to look at the one thing they all have in common that is related to what they are applying for.

Actually one of the major points brought up by the INCUS convention is that URM differences do continue to exist in the Step exams and residency match. One of the positive side effects of a pass/fail switch would be that way, way more minority med students would now stand a chance at surgical specialty matches.

Well... I mean at some point we have to acknowledge reality and recognize that when we accept people with consistently lower stats we shouldn't be surprised when that same cohort performs lower than the other groups of people the rest of the way through. Looking at large population groups of course. At some point you have to actually compete with your peers directly and be compared directly to them. No one is entitled to a residency in X competitive specialty.

Like I have said, I'm not necessarily opposed to a change in the process, I just think the current proposals are simply reactionary and will cause far more problems than they will fix.

---

Members don't see this ad.

 

[Lawpy]

Actually one of the major points brought up by the INCUS convention is that URM differences do continue to exist in the Step exams and residency match. One of the positive side effects of a pass/fail switch would be that way, way more minority med students would now stand a chance at surgical specialty matches.

Also, why does this conversation always pretend the USMLE has always been like this??? Go back to the 90s and nobody cared about your score, go back any further and the score literally doesn't exist. It's not like they had a hard time identifying highly competent students to fill desirable residency slots twenty years ago.

I mean a decade ago, you needed a low 30 MCAT to go to top schools...

Also Step 2 CI?

 

[deleted804295]

I did well on both (equivalent to 524+ on MCAT and a 250+ on USMLE).

The difficulty of the MCAT comes from testing your academic aptitude, rather than your knowledge base. You could study half as much as someone else and beat them by a lot, because the upper end of the bell curve is mostly differentiated by critical thinking/reasoning about new information presented to you in science and verbal passages. An example would be a passage describing the results of an experiment and asking you to interpret them.

The USMLE is an entirely different beast that is first and foremost testing your knowledge base / how much you memorized. Someone that memorized twice as much of First Aid will always score higher than someone who only knows half of it. An example would be a question describing a specific virus, that then asks you whether that virus is a DNA virus, positive sense RNA, or negative sense RNA. Theres no way to really reason out a correct answer, you either know it or you dont.

There are minor exceptions on both exams - the MCAT has a few discrete questions that are pure recall and the USMLE does have a few experiment interpretation questions - but overall that's the clear difference in flavor.

So whether you think it's easy or hard depends on where your strength lies. If you got a top percentile MCAT the USMLE is probably going to feel much worse to prepare for because its largely stuffing huge volumes of rote recall into your head. On the flip side if the MCAT felt extremely hard to study for and score well on, you'll probably find the USMLE feels a lot more fair, because theres a much more direct connection between studying more hours --> learning more information --> scoring higher.

Overall the correlation between exams is about 0.60. You can find many posts online about people who struggled a lot with the MCAT, but that did hundreds of thousands of Anki flashcards to memorize the entirety of First Aid and easily hit 250+ on USMLE.

This sounds pleasant for me a master memorizer.

 

[piii]

Step 1 was more stressful and crazy test than MCAT. Step 2 was way worse than step 1

 

[deleted804295]

I dont want them messing with step at all in the next 4 years. They can mess with it if I become a med student and *after* I take it I'm fine playing the game of the system and by changing things without incremental changes people might have their futures screwed.

 

[notretired]

Well... I mean at some point we have to acknowledge reality and recognize that when we accept people with consistently lower stats we shouldn't be surprised when that same cohort performs lower than the other groups of people the rest of the way through. Looking at large population groups of course. At some point you have to actually compete with your peers directly and be compared directly to them. No one is entitled to a residency in X competitive specialty.

Like I have said, I'm not necessarily opposed to a change in the process, I just think the current proposals are simply reactionary and will cause far more problems than they will fix.
[/QUOTE]

I'm torn here because it's ridiculous to think that the oft quoted competitive specialties somehow require more knowledge than the others; so the step score cut-offs etc. seem pretty arbitrary in regards to who they get vs. who they need. With that said, the system is simply reflective of other competitive career fields in this respect. I don't believe anyone is entitled to anything; if it is a competition then objective measures must remain in place and it is on applicants to stratify themselves.

 

[efle]

Yeah that's not really what the argument is though, and there is definitely a difference between a 255 and say a 230. I agree USMLE is not good at separating out the top end of the bell curve, but honestly I don't really think that matters as much as people think it does. You also happen to be surrounded by mostly people at the right end of the bell curve so I think your viewpoint is skewed by where you are training.


But does it though? You go to Hopkins if I'm not mistaken. Do the people with 240s not match competitive specialties and the people with 250s do? I doubt it. This often gets thrown around but the charting outcomes and match data say otherwise. Step scores are important no doubt but not getting a 250 isn't a death sentence for ANY specialty. I think the importance is overstated due to medical student neuroticism. People with 240s, and even 230s, match competitive specialties at high rates. If you have a 240 and the rest of your application is in order then you have an excellent chance of matching X specialty of your choice.



I mean better on the wards as in they give better and more concise presentations to the attendings, they can get assigned a new patient and take a quality history and do a solid exam, they are constantly creating realistic and high quality differentials on their patients. Essentially they do the exact same things as any other student but they do so much more effectively than everyone else. You know those students that get told by attendings, "I'm impressed with how you think, you already approach patients like an intern should" midway through 3rd year? Those people almost always are people who did very well on boards.

As to the bolded I don't think anyone is really arguing that. Standardized tests are always just a mediocre way of assessing stuff like that, but USMLE is actually roughly correlated to board pass rates and the content is honestly pretty dang relevant to medicine. So when you have large swaths of people applying to residency programs that all look the exact same you need to look at the one thing they all have in common that is related to what they are applying for.



Well... I mean at some point we have to acknowledge reality and recognize that when we accept people with consistently lower stats we shouldn't be surprised when that same cohort performs lower than the other groups of people the rest of the way through. Looking at large population groups of course. At some point you have to actually compete with your peers directly and be compared directly to them. No one is entitled to a residency in X competitive specialty.

Like I have said, I'm not necessarily opposed to a change in the process, I just think the current proposals are simply reactionary and will cause far more problems than they will fix.

You can look at the stratified data on the tableau page, and the match rates for surgical subspecialties are indeed different for 230s vs 250s. To use ENT as an example, the match rate for 230s was 74% and for 250s was 92%. So while it's technically true that you could apply ENT with an average step1 and probably match... that's a daunting enough failure rate that unless someone knew they had a spot waiting for them at their home institution, many or most would rather just apply for something else. And I agree that big name schools are an exception where it's much easier to match well in spite of a low score, but the data I'm looking at here is national.

We'll just have to agree to disagree about whether Step1 score correlates to how well you handle presenting on rounds or being a good member of the team. If anything my experience so far would've been an inverse correlation, where the super bookish people that did a quarter million anki cards and nailed a 260 are actually more awkward and less confident at the bedside. But we're in the realm of anecdote and speculation here. I also agree that we shouldn't be surprised about URM match data, and if we wanted to address it, the better approach would be upstream at the point of medical admissions.

But what's your take on how this impacts preclinical education - are you OK with the dystopian version of the year 2025, where the first half of earning an MD is really just a competition to see who can do the best job of maturing a 30,000 card anki deck? To me, that looks like where things are headed. That's another key difference from the MCAT, which nobody really starts studying for 12-24 months ahead of time the way people aiming for 250+ usually do for Step1. The MCAT doesn't threaten to override the college curriculums of the country in the way the Step currently is for medical school coursework.

 

[efle]

I'm torn here because it's ridiculous to think that the oft quoted competitive specialties somehow require more knowledge than the others; so the step score cut-offs etc. seem pretty arbitrary in regards to who they get vs. who they need. With that said, the system is simply reflective of other competitive career fields in this respect. I don't believe anyone is entitled to anything; if it is a competition then objective measures must remain in place and it is on applicants to stratify themselves.

This. Quoted for truth. It's at least slightly defensible to say that law schools or med schools or colleges with their pick of the litter have good logic for favoring the highest LSAT/MCAT/SAT scores.

There's honestly no reason at all for an ENT attending to care whether their new trainee knew which step of the citric acid cycle produces GTP.

That's a particularly egregious example, but there really are a lot of questions like that on the exam, and y'all get my point.

 

[Lawpy]

This. Quoted for truth. It's at least slightly defensible to say that law schools or med schools or colleges with their pick of the litter have good logic for favoring the highest LSAT/MCAT/SAT scores.

There's honestly no reason at all for an ENT attending to care whether their new trainee knew which step of the citric acid cycle produces GTP.

That's a particularly egregious example, but there really are a lot of questions like that on the exam, and y'all get my point.

If Step 1 were made more aptitude based like the MCAT, it resolves a lot of problems focusing on memorization/Anki craze without having to change the score reporting. And aptitude tests are way better anyways especially seeing how critical the MCAT is

 

[Lawpy]

Also i ask about Step 2 CI because that exam is apparently more practical and relevant than Step 1 and it doesnt affect clinical curricula

 

[notretired]

But what's your take on how this impacts preclinical education - are you OK with the dystopian version of the year 2025, where the first half of earning an MD is really just a competition to see who can do the best job of maturing a 30,000 card anki deck? To me, that looks like where things are headed. That's another key difference from the MCAT, which nobody really starts studying for 12-24 months ahead of time the way people aiming for 250+ usually do for Step1. The MCAT doesn't threaten to override the college curriculums of the country in the way the Step currently is for medical school coursework.

I mean if the schools just disintegrated and medicine suddenly became an open opportunity to anyone who could hack a 240+ step score - is that really the worst thing the world? Basically the purest form of a meritocracy: students interview directly with residency programs for a two year internship; those asked to "stay on" start residency at the end. Students are already putting focus on step at the expense of the med school coursework, is it really making them worse doctors? I don't think we need to fight to turn the preclinical years into something they are not - just grind through them as quickly as possible and get into a clinical environment. We don't need the schools for anything really.

 

[efle]

If Step 1 were made more aptitude based like the MCAT, it resolves a lot of problems focusing on memorization/Anki craze without having to change the score reporting. And aptitude tests are way better anyways especially seeing how critical the MCAT is

But we gotta remember this is the USA medical licensing exam, it's fundamentally supposed to be a check on our knowledge to make sure we've learned the minimum information we need. If it became more like the MCAT/LSAT, you'd be licensing physicians based on their intelligence instead of their studies.

 

[notretired]

If Step 1 were made more aptitude based like the MCAT, it resolves a lot of problems focusing on memorization/Anki craze without having to change the score reporting. And aptitude tests are way better anyways especially seeing how critical the MCAT is

What is the value in transitioning Step to an aptitude test? The entire idea is to test student's knowledge/work ethic over the preclinical years. I would argue that correlations btw good step and good resident alluded to above could very well be born out of the fact that both require tons of hard work. I don't think we need another faux 7+ hr "IQ" test to add to the mix when we already have SAT's, mcat, etc. I also don't really think the mcat is as much of an "aptitude" test as many claim.

 

[Lawpy]

But we gotta remember this is the USA medical licensing exam, it's fundamentally supposed to be a check on our knowledge to make sure we've learned the minimum information we need. If it became more like the MCAT/LSAT, you'd be licensing physicians based on their intelligence instead of their studies.

You can have both though. See MCAT

Also Step 2 CI? I'm not sure where you found CI data

 

[efle]

I mean if the schools just disintegrated and medicine suddenly became an open opportunity to anyone who could hack a 240+ step score - is that really the worst thing the world? Basically the purest form of a meritocracy: students interview directly with residency programs for a two year internship; those asked to "stay on" start residency at the end. Students are already putting focus on step at the expense of the med school coursework, is it really making them worse doctors? I don't think we need to fight to turn the preclinical years into something they are not - just grind through them as quickly as possible and get into a clinical environment. We don't need the schools for anything really.

I personally think the current medical school admissions process is a much better way of selecting physicians than the step1 exam would be. But I agree that for teaching [Topic XYZ] it's probably fine to have students learn it from Pathoma or Boards&Beyond lectures instead of a prof at their school.

 

[efle]

I also don't really think the mcat is as much of an "aptitude" test as many claim.

It's pretty hard to explain how someone could leaf through a review book for a few weeks, and then score 100th percentile, unless it's mostly a test of interpreting novel information given in the passages themselves. People also talk about how a privileged high school + prep classes can give someone an Ivy League SAT score, but in reality the studies show that's nonsense, and the public school kid who strolls in with zero prep and hits top 1% is just plain smart.

You can have both though. See MCAT

Also Step 2 CI? I'm not sure where you found CI data

Haven't seen the step2 CI data but it'll be at the bottom of your score report. What year are you right now, M2? Taking step1 soon?

 

[premed9901]

The USMLE is an entirely different beast that is first and foremost testing your knowledge base / how much you memorized. Someone that memorized twice as much of First Aid will always score higher than someone who only knows half of it.

If you want to be a surgical sub specialist (e.g. orthopedics, neurosurg, plastics, ophthalmology, urology, ENT, etc) you need to beat 80-90% of US medical students, which is VERY hard and stressful.

It's very possible by the time you guys are taking the Step 1, there wont be any such thing as a 250+ target for competitive specialties any more.

So the lower scorer only has themselves to blame? I can see how a relatively weaker student at a top school would want step to become pass/fail.

Can you see how this change would be extremely frustrating for those students that chose a lower-tier school for financial reasons?

 

[notretired]

I personally think the current medical school admissions process is a much better way of selecting physicians than the step1 exam would be. But I agree that for teaching [Topic XYZ] it's probably fine to have students learn it from Pathoma or Boards&Beyond lectures instead of a prof at their school.

Why do you say this? I think Residency programs would do an equal if not better job screening candidates in interviews based on the fact that they are still heavily involved with clinical medicine - not to mention the noncommittal two year internship by which they could further screen. For interns that get axed before starting residency, they will be in debt to the tune of maybe $2k spent on step prep materials? That's a bit better than $300+k.

 

[efle]

So the lower scorer only has themselves to blame? I can see how a relatively weaker student at a top school would want step to become pass/fail.

Can you see how this change would be extremely frustrating for those students that chose a lower-tier school for financial reasons?

Depends what you consider blame-worthy. Suppose someone chooses to devote themselves to their school curriculum, which consists of material their professors would actually want their trainees to know on the wards. It skips most of the useless rote details that nobody would ever need in a hospital. Dedicated rolls around and uh-oh, your starting point is a 160 because what you spent two years studying from your attendings that will train you on rotations is very different than what the USMLE tests. You bust your ass for 6 weeks but can only get up to a 225. Does this person deserve to have their dream of [insert surgical specialty] denied to them?

Do you pity the public school students of yesteryear? Was there a great injustice done to the 90% of medical students not at top-ranked schools in all the years prior to the new millennium?

Looking at the numbers, the reality is that top schools fill very few slots in any given area. Hopkins sent a whopping 3 people into dermatology last year. School name really doesn't much matter for the typical derm applicant, you aren't going to lose your spot because of big names if it switches to quartiles.

 

[efle]

Why do you say this? I think Residency programs would do an equal if not better job screening candidates in interviews based on the fact that they are still heavily involved with clinical medicine - not to mention the noncommittal two year internship by which they could further screen. For interns that get axed before starting residency, they will be in debt to the tune of maybe $2k spent on step prep materials? That's a bit better than $300+k.

Residency programs currently rely a lot on their applicant pool already consisting of a heavily filtered group of people: those who made it into US medical schools.

If it became some kind of open application system where 100,000+ people per year could take the step and apply (which is the current number taking the MCAT annually), residencies would be overwhelmed. It's already the case in competitive specialties that they have to filter out with a step cutoff. Can't even imagine what it would be like if your hospital with 10 spots got thousands of applicants. It would be impossible to meaningfully review any significant chunk of them. It's already a big challenge for medical schools, who have recently started adopting analogous MCAT cutoff screens for the same reason.

As @Lucca frequently mentions, if anything a step in the opposite direction might help things, by placing a limit on the numbers of residencies that one is allowed to apply to.

 

[Lawpy]

From what i'm seeing, i think Step 2 has lower CI than Step 1 but i'm not sure. And Step 2 doesnt affect clinical curricula while being relevant and useful. But no idea if its aptitude based

 

[notretired]

It's pretty hard to explain how someone could leaf through a review book for a few weeks, and then score 100th percentile, unless it's mostly a test of interpreting novel information given in the passages themselves. People also talk about how a privileged high school + prep classes can give someone an Ivy League SAT score, but in reality the studies show that's nonsense, and the public school kid who strolls in with zero prep and hits top 1% is just plain smart.

I think you're forgetting the background you had prior to taking the test. Someone starting from scratch (i.e. non-science background) has a long road to hoe for a top %ile mcat score. I know this because I did it. To say that I could've reasoned my way through passages to a 99%ile score before taking pre-reqs is absolutely laughable. Also kahn academy has a 130pt improvement avg to those who complete their free SAT course. Some my reach a ceiling faster than others - but that's a pretty significant improvement curve for a test based on "plain smarts."

 

[premed9901]

Depends what you consider blame-worthy. Suppose someone chooses to devote themselves to their school curriculum, which consists of material their professors would actually want their trainees to know on the wards. It skips most of the useless rote details that nobody would ever need in a hospital. Dedicated rolls around and uh-oh, your starting point is a 160 because what you spent two years studying from your attendings that will train you on rotations is very different than what the USMLE tests. You bust your ass for 6 weeks but can only get up to a 225. Does this person deserve to have their dream of [insert surgical specialty] denied to them?

Do you pity the public school students of yesteryear? Was there a great injustice done to the 90% of medical students not at top-ranked schools in all the years prior to the new millennium?

Looking at the numbers, the reality is that top schools fill very few slots in any given area. Hopkins sent a whopping 3 people into dermatology last year. School name really doesn't much matter for the typical derm applicant, you aren't going to lose your spot because of big names if it switches to quartiles.

You're implying preclinical stuff= useful for wards, while step 1 stuff= not useful for wards. I disagree completely. Also, LCME and pretty standard stuff covered and whatnot.

And the reality is that school name matters a lot for competitive specialties. Top 40 schools accounted for 40% of all Derm, Plastics, Neurosurgery spots, with top 10 I imagine being even higher. Just because Johns hopkins only sent 3 people to derm (~3% of the class which is still higher than my schools 1%) doesn't mean other top schools don't send a disproportionately large amount of students to competitive specialties (Like WashU sending 9 out of 125, or Harvard or NYU etc. )not explained by step score differences.

 

[efle]

I think you're forgetting the background you had prior to taking the test. Someone starting from scratch (i.e. non-science background) has a long road to hoe for a top %ile mcat score. I know this because I did it. To say that I could've reasoned my way through passages to a 99%ile score before taking pre-reqs is absolutely laughable. Also kahn academy has a 130pt improvement avg to those who complete their free SAT course. Some my reach a ceiling faster than others - but that's a pretty significant improvement curve for a test based on "plain smarts."

Oh I don't mean anyone could take it without any science background. I mean among people who have taken the prereqs and then take the test, what differentiates the top of the curve isn't that they studied longer or knew more.

Interesting to hear that advertisement from Khan, they must not have been included in the review study commissioned by the College Board I read a while back. It showed the max affect of prep classes was about 40 pts, barely significant p. And even if we grant ~150 pts, it's pretty damn silly to pretend the kid with a 2350 only got into an elite college because his parents could afford a class. Take away his 150 free points, and you're still left trying to explain why he's beating 99% of the rest. Big hint being that again, it's not got anything to do with recall or knowledge.

I say all this as someone who went to their public high school and couldn't afford a prep class. My MCAT review was also just some out of date Berkeley Review books I bought cheap on craigslist. I don't want to come off as a privileged ass protecting my ego.

 

[notretired]

Residency programs currently rely a lot on their applicant pool already consisting of a heavily filtered group of people: those who made it into US medical schools.

If it became some kind of open application system where 100,000+ people per year could take the step and apply (which is the current number taking the MCAT annually), residencies would be overwhelmed. It's already the case in competitive specialties that they have to filter out with a step cutoff. Can't even imagine what it would be like if your hospital with 10 spots got thousands of applicants. It would be impossible to meaningfully review any significant chunk of them. It's already a big challenge for medical schools, who have recently started adopting analogous MCAT cutoff screens for the same reason.

As @Lucca frequently mentions, if anything a step in the opposite direction might help things, by placing a limit on the numbers of residencies that one is allowed to apply to.

They'd be interviewing the same number of applicants they are now because they'd be auto-screening apps the same way they are now. Only difference is that applicants won't be in crippling debt and the doors would be open to anyone who could afford the cost of step and access to a public library.

 

[efle]

You're implying preclinical stuff= useful for wards, while step 1 stuff= not useful for wards. I disagree completely. Also, LCME and pretty standard stuff covered and whatnot.

And the reality is that school name matters a lot for competitive specialties. Top 40 schools accounted for 40% of all Derm, Plastics, Neurosurgery spots, with top 10 I imagine being even higher. Just because Johns hopkins only sent 3 people to derm (~3% of the class which is still higher than my schools 1%) doesn't mean other top schools don't send a disproportionately large amount of students to competitive specialties (Like WashU sending 9 out of 125, or Harvard or NYU etc. )not explained by step score differences.

There are schools that teach to the boards (a classic example being Mizzou which revamped their curriculum to be boards oriented and shot up to a 240 average), and similar schools that don't (take Maryland as an example I'm familiar with, that has an average in the 220s). It's a Venn diagram with a lot of overlap, not a perfect distinction, but if you think your best medicine attending could sit down and ace a block of step1 Uworld tonight, you're living on a different planet than me.

You're crediting all that representation to school name without controlling for anything else. The MCAT and Step1 have a .60 correlation after all. You take the ~1000-2000 students at top ranking schools and look at their Step1 distribution, it's probably the source of most of their over-representation.

 

[efle]

They'd be interviewing the same number of applicants they are now because they'd be auto-screening apps the same way they are now. Only difference is that applicants won't be in crippling debt and the doors would be open to anyone who could afford the cost of step and access to a public library.

The new AAMC residency finder tool shows data on numbers interviewed and matched. At highly competitive stuff (like major west coast Derm) it's about 10% being interviewed and of that, 1-3% match.

That's with a few hundred apps. If that grew to many thousands, like what we see for a lot of med schools right now, they'd have to automatically screen all but a few percent of apps to actually read. The insane emphasis on step1 (with its garbage confidence interval) would only get way worse.

 

[notretired]

Oh I don't mean anyone could take it without any science background. I mean among people who have taken the prereqs and then take the test, what differentiates the top of the curve isn't that they studied longer or knew more.

Interesting to hear that advertisement from Khan, they must not have been included in the review study commissioned by the College Board I read a while back. It showed the max affect of prep classes was about 40 pts, barely significant p. And even if we grant ~150 pts, it's pretty damn silly to pretend the kid with a 2350 only got into an elite college because his parents could afford a class. Take away his 150 free points, and you're still left trying to explain why he's beating 99% of the rest. Big hint being that again, it's not got anything to do with recall or knowledge.

I say all this as someone who went to their public high school and couldn't afford a prep class. My MCAT review was also just some out of date Berkeley Review books I bought cheap on craigslist. I don't want to come off as a privileged ass protecting my ego.

I was wrong it was 115pts with 20hrs of studying - and the study was done in conjunction with college board - not uniform amongst all groups tho. I didn't read thru the study just looked up the quick blurb posted below. Feel free to scrutinize further. I commend you for taking the path you did as my background was very similar; but my own anecdotal experiences have led me to believe that performance on these tests is not nearly as predetermined as we make it out to be.

Studying for the SAT for 20 hours on Khan Academy associated with 115-point average score increase

We’re excited to announce today that studying for the SAT for 20 hours on Khan Academy’s free Official SAT Practice is associated with an average score gain of 115 points. That’s nearly double the average score gain compared to students who don’t use our free test prep. Official SAT Practice is...

www.khanacademy.org

 

[Splenda88]

This is when I took CK...

This score is determined by your overall performance on Step 2 CK. For administrations between July 1, 2016 and June 30, 2017, the mean and standard deviation for first-time examinees from U.S. and Canadian medical schools were approximately 242 and 17, respectively, with most scores falling between 190 and 270. A score of 209 is set by USMLE to pass Step 2 CK. The standard error of measurement (SEM)‡ for this scale is six points.

 

[notretired]

The new AAMC residency finder tool shows data on numbers interviewed and matched. At highly competitive stuff (like major west coast Derm) it's about 10% being interviewed and of that, 1-3% match.

That's with a few hundred apps. If that grew to many thousands, like what we see for a lot of med schools right now, they'd have to automatically screen all but a few percent of apps to actually read. The insane emphasis on step1 (with its garbage confidence interval) would only get way worse.

Right but the programs aren't reading through all of those apps - they employ hard screens; which would give them the same applicant pool to consider for interviews + a few more that come out of the woodwork due to the significantly lessened cost burden. Med schools do the exact same thing now with mcat scores.

 

[efle]

Right but the programs aren't reading through all of those apps - they employ hard screens; which would give them the same applicant pool to consider for interviews + a few more that come out of the woodwork due to the significantly lessened cost burden. Med schools do the exact same thing now with mcat scores.

Ah I get what you're saying, the same people would end up getting their apps reviewed.

I guess I'd see the more likely version to be a jump straight into M3, since it's still useful to have the ~2 years of rotating throughout specialties before choosing what you want to do and going into residency training for it. Something like people take a gap year after college to study up for Step 1, take it, and then apply to "med school" where med school is just the latter half of what we do now. Saves half the money, at least.

Maybe that's what will exist a generation from now, but it's really only been ~5 years that streamed packages like Pathoma or Boards&Beyond have been available to supplant professors in a lecture hall. Give it 50 years and then we'll bump this thread.

 

[notretired]

Ah I get what you're saying, the same people would end up getting their apps reviewed.

I guess I'd see the more likely version to be a jump straight into M3, since it's still useful to have the ~2 years of rotating throughout specialties before choosing what you want to do and going into residency training for it. Something like people take a gap year after college to study up for Step 1, take it, and then apply to "med school" where med school is just the latter half of what we do now. Saves half the money, at least.

Maybe that's what will exist a generation from now, but it's really only been ~5 years that streamed packages like Pathoma or Boards&Beyond have been available to supplant professors in a lecture hall. Give it 50 years and then we'll bump this thread.

Agreed, was a fun discussion nonetheless haha

 

[TheIllusionist]

I took it in May. I'll say it had a lot less buzzwords and pure regurgitation than older practice NBMEs, for sure. But it was still much, much more a test of knowledge than reasoning.

When you get a question wrong in something like physics or organic chemistry, it's "oh yeah, that makes sense, I see why I'm wrong and that's right."

With the usmle, even this year's latest edition of new practice NBMEs, it's always: "Well that's a factoid I'd just never learned before. Guess I should spend an hour memorizing which CYP subtypes metabolize each major drug class."

I agree with this assessment. I did better than you on the MCAT and worse on the USMLEs. The MCAT, particularly VR, had a lot of aptitude component you could reason your way through. There's plenty of people that, without dedicated studying, could pull off a 30 (~508) on the MCAT having just taken the prerequisites but not really having any dedicated study time. It's been 11 years, and I'm fairly certain if you gave me a couple weeks to review my physics formulas and my ochem I could probably pull off a respectable MCAT score even today.

There's no one that could ever pull off an equivalent percentile on the USMLE (240+) without study resources explicitly dedicated to the task. The knowledge base is just too large and the specific recall component is too strong. They're trying to make it more reasoning based but it's reasoning based on nit picked facts. It's only been 8 years, but I'm pretty sure if you were to give me a few weeks to review, *might* barely pass Step 1. Hell, I might fail without a month or more dedicated time. Step 2 and Step 3 are much more clinically relevant and less focused on the minutia, so I'd probably do very well on that if you gave me a few weeks to review my peds and obstetrics (I'm an adult subspecialist).

(Same thing with the SAT - I'm fairly certain I could get a similar score today to what I got 15 years ago - might need to spend a few days just dusting out cobwebs for algebra, but I remember that better than I remember specific enzymes for biochem)

yeah SAT was a bigger strength for me. Got 2330 or a 1540 equivalent today I think. But way way different test pool and many kids didn't try then so 99.9th percentile means less for it.

I started at like a 2050 and took a bunch of practice tests, and it went up a lot. I have seen people do it on the supposedly super IQ heavy LSAT. I know 3 personally who went from 150s to 170s. One was my college debate partner who eventually got a 178. But the reliability of who beats the aptitude tests with more time is less so thab with usmle. But with USMLE. More quality passes of first aid, zanki, and u world info almost always means higher scores.

 

[efle]

yeah SAT was a bigger strength for me. Got 2330 or a 1540 equivalent today I think. But way way different test pool and many kids didn't try then so 99.9th percentile means less for it.

I can add a couple more n. Myself and a buddy were both top 0.1% on the MCAT and both barely cracked 90th percentile on the Step1. The top end of the curve rewards entirely different things on the USMLE (namely, hundreds of thousands of anki reviews).

In fact, I don't know a single person who hit 260+ this year without anki, which neither of us used.

 

[TheIllusionist]

I can add a couple more n. Myself and a buddy were both top 0.1% on the MCAT and both barely cracked 90th percentile on the Step1. The top end of the curve rewards entirely different things on the USMLE (namely, hundreds of thousands of anki reviews).

In fact, I don't know a single person who hit 260+ this year without anki, which neither of us used.

yeah I wish I used Zanki more. I was acually 99th percentile on IM, neuro, psych, and pes shelves so my step2ck was an underperform. It sucked just to forget stuff and see mcqs on my exam that I knew for sure I knew the answers to in the past.straight up OCPD memory contest . why some carib kids who sre awful at all other past tests can score over 260

 

[907914]

along with students who chose their state school for financial reasons

Out of all that, the one thing I picked up was this - I don't think that students who chose their state school for financial reasons will be disadvantaged by p/f step1. Literally the only schools that will see a bump in preference if this happens are also the schools that offer the best and most comprehensive financial aid. Outside of T10/T20 schools, financial aid is almost always the same and your state school is probably the best option regardless.

 

[TheIllusionist]

As much as I love Goljan, I hated when he said memorization was bad for the USMLE. He gave specific two examples

How kids memorized horseshoe kidney for turners, but now they were testing "thinking" with inferior mesenteric artery correlate

How they used to test troponins are renally cleared on IM boards so could remain elevated longer than usual but were now testing on STEP1

I mean I love the guy because I think he is an awesome person and educator and yes there is some element of thinking but come on. Even his examples were just memorization extensions

 

[Splenda88]

The MCAT, as someone said above, is reading comprehension exam... It favors people that have very strong English/Humanities background. These people can pretty much get 20-24 out of 30 in BS/VR without lifting a finger. Even the BS section when I took that nonsense was a reading comprehension exercise. I spent 60-70% of my preparation time in Verbal Reasoning just to get 7/15.


I think Step 1/2 are easier tests than MCAT.

 

[Raryn]

The MCAT, as someone said above, is reading comprehension exam... It favors people that have very strong English/Humanities background. These people can pretty much get 20-24 out of 30 in BS/VR without lifting a finger. Even the BS section when I took that nonsense was a reading comprehension exercise. I spent 60-70% of my preparation time in Verbal Reasoning just to get 7/15.


I think Step 1/2 are easier tests than MCAT.

Not easier. Different.

MCAT for me was a LOT easier than Step 1, but plenty of other people in this thread say the opposite. Some people find it easier to digest and regurgitate the huge amount of information you need for the USMLE. Some people find it easier to perform the critical thinking tasks you need for the MCAT.

 

[efle]

yeah I wish I used Zanki more. I was acually 99th percentile on IM, neuro, psych, and pes shelves so my step2ck was an underperform. It sucked just to forget stuff and see mcqs on my exam that I knew for sure I knew the answers to in the past.straight up OCPD memory contest . why some carib kids who sre awful at all other past tests can score over 260

Yeah the step1 subreddit seems to have a post every week about someone who barely managed a competitive MCAT and hit 260+, and invariably they credit a combination of Zanki/Brosencephalon and additional Qbanks (Kaplan, Qmax).

The days of only studying UFAPS during dedicated for a 260 are long gone. Now it's anki for a year + entire additional Qbanks prior to dedicated, then UFAPS as review, that are the norm for high scores.

 

[efle]

Out of all that, the one thing I picked up was this - I don't think that students who chose their state school for financial reasons will be disadvantaged by p/f step1. Literally the only schools that will see a bump in preference if this happens are also the schools that offer the best and most comprehensive financial aid. Outside of T10/T20 schools, financial aid is almost always the same and your state school is probably the best option regardless.

QFT again. For both undergrad and medical school, the well-endowed private schools offered giant need-based scholarships and were actually cheaper options for me than four years at my state schools.

 

[Lawpy]

I can add a couple more n. Myself and a buddy were both top 0.1% on the MCAT and both barely cracked 90th percentile on the Step1. The top end of the curve rewards entirely different things on the USMLE (namely, hundreds of thousands of anki reviews).

In fact, I don't know a single person who hit 260+ this year without anki, which neither of us used.

yeah I wish I used Zanki more. I was acually 99th percentile on IM, neuro, psych, and pes shelves so my step2ck was an underperform. It sucked just to forget stuff and see mcqs on my exam that I knew for sure I knew the answers to in the past.straight up OCPD memory contest . why some carib kids who sre awful at all other past tests can score over 260

Yeah the step1 subreddit seems to have a post every week about someone who barely managed a competitive MCAT and hit 260+, and invariably they credit a combination of Zanki/Brosencephalon and additional Qbanks (Kaplan, Qmax).

The days of only studying UFAPS during dedicated for a 260 are long gone. Now it's anki for a year + entire additional Qbanks prior to dedicated, then UFAPS as review, that are the norm for high scores.

That's why I'm pushing for Step 1 to be more of an aptitude test. If the MCAT exam tests both intelligence and content mastery, why can't the same be for Step exams? That way, preclinical curricula won't be blindly tossed out the window, the Anki craze will stop, and high scores will truly be indicative of both content mastery and excellent reasoning skills.

You don't have to change the scoring. Just redesign the test completely to make it aptitude based.

 

[efle]

That's why I'm pushing for Step 1 to be more of an aptitude test. If the MCAT exam tests both intelligence and content mastery, why can't the same be for Step exams? That way, preclinical curricula won't be blindly tossed out the window, the Anki craze will stop, and high scores will truly be indicative of both content mastery and excellent reasoning skills.

You don't have to change the scoring. Just redesign the test completely to make it aptitude based.

Because again, this is a licensure exam meant to function as a confirmation that we've learned enough knowledge from our first two years of studies. You absolutely do NOT want this to be an exam where someone can make up for a lack of critical knowledge via sheer brainpower, because you absolutely do not want to go see a physician for care that has a terrible knowledge base and made up for it by being brilliant at multiple choice exams.

It's like saying the bar exam for lawyers should be changed because being smart is just as important as knowing about past rulings or current laws. That's just not an option. You want a smart lawyer? Go hire a guy who was top of class at a top law school. When you see someone passed the bar, it has to mean they know about the laws. Doesn't matter how smart they are. Doesn't matter where they learned it. It's a stamp of approval from the national governing body that they learned the information they need to practice law, and it's not safe to chisel away at that by adding more and more aptitude questions.

 

[Lawpy]

Because again, this is a licensure exam meant to function as a confirmation that we've learned enough knowledge from our first two years of studies. You absolutely do NOT want this to be an exam where someone can make up for a lack of critical knowledge via sheer brainpower, because you absolutely do not want to go see a physician for care that has a terrible knowledge base and made up for it by being brilliant at multiple choice exams.

It's like saying the bar exam for lawyers should be changed because being smart is just as important as knowing about past rulings or current laws. That's just not an option. You want a smart lawyer? Go hire a guy who was top of class at a top law school. When you see someone passed the bar, it has to mean they know about the laws. Doesn't matter how smart they are. Doesn't matter where they learned it. It's a stamp of approval from the national governing body that they learned the information they need to practice law, and it's not safe to chisel away at that by adding more and more aptitude questions.

But you can have an exam that tests both content review and reasoning skills? Like it's pretty much impossible to ace the MCAT just by reasoning your way through without having any content review. The Step exams can be the same way. Right now, it's looking like memorizing = key to getting super high scores. Why not change that a bit to shift more heavily on reasoning skills?

Like passing Step 1 can still be done easily with just content review. But content review alone should no longer be sufficient to score 250+, just like content review alone isn't enough to score a 520+. But reasoning skills alone aren't enough to score that high either. Both are needed.

 

[notretired]

But you can have an exam that tests both content review and reasoning skills? Like it's pretty much impossible to ace the MCAT just by reasoning your way through without having any content review. The Step exams can be the same way. Right now, it's looking like memorizing = key to getting super high scores. Why not change that a bit to shift more heavily on reasoning skills?

Like passing Step 1 can still be done easily with just content review. But content review alone should no longer be sufficient to score 250+, just like content review alone isn't enough to score a 520+. But reasoning skills alone aren't enough to score that high either. Both are needed.

And the purpose of this would be?

 

[efle]

But you can have an exam that tests both content review and reasoning skills? Like it's pretty much impossible to ace the MCAT just by reasoning your way through without having any content review. The Step exams can be the same way. Right now, it's looking like memorizing = key to getting super high scores. Why not change that a bit to shift more heavily on reasoning skills?

Like passing Step 1 can still be done easily with just content review. But content review alone should no longer be sufficient to score 250+, just like content review alone isn't enough to score a 520+. But reasoning skills alone aren't enough to score that high either. Both are needed.

But it's not the high end you need to worry about, it's the low end. This is a licensure test and the scary outcome isn't that smart people get their scores held back. The scary outcome is that smart people who don't know anything pass it off aptitude alone and get licensed.

So, this test fundamentally should not feel anything like the MCAT or SAT. Those are tests of your ability to quickly learn and apply. They say you're able to be a good student.

For the bar exam or USMLE you don't want to identify who is able to learn. You need to see who actually did the learning. Whether it took them 100 hours or 10,000 hours, your #1 priority is to make sure that people can't pass it without having learned the knowledge they need to practice.

 

[Splenda88]

Not easier. Different.

MCAT for me was a LOT easier than Step 1, but plenty of other people in this thread say the opposite. Some people find it easier to digest and regurgitate the huge amount of information you need for the USMLE. Some people find it easier to perform the critical thinking tasks you need for the MCAT.

Did/do you have a very strong English/Humanities background?

 

[Lawpy]

But it's not the high end you need to worry about, it's the low end. This is a licensure test and the scary outcome isn't that smart people get their scores held back. The scary outcome is that smart people who don't know anything pass it off aptitude alone and get licensed.

So, this test fundamentally should not feel anything like the MCAT or SAT. Those are tests of your ability to quickly learn and apply. They say you're able to be a good student.

For the bar exam or USMLE you don't want to identify who is able to learn. You need to see who actually did the learning. Whether it took them 100 hours or 10,000 hours, your #1 priority is to make sure that people can't pass it without having learned the knowledge they need to practice.

I'm a little lost, so help me out here. Step 1 right now is focusing heavily on memorizing stuff. And getting high scores require memorizing more of the content than needed. Do you believe that someone who just barely passed Step 1 has the necessary knowledge base to function clinically well?

 

[notretired]

I'm a little lost, so help me out here. Step 1 right now is focusing heavily on memorizing stuff. And getting high scores require memorizing more of the content than needed. Do you believe that someone who just barely passed Step 1 has the necessary knowledge base to function clinically well?

According to the licensing board they do; that's the entire point of the test. Your own standards and opinions not withstanding.

 

[Lawpy]

But it's not the high end you need to worry about, it's the low end. This is a licensure test and the scary outcome isn't that smart people get their scores held back. The scary outcome is that smart people who don't know anything pass it off aptitude alone and get licensed.

So, this test fundamentally should not feel anything like the MCAT or SAT. Those are tests of your ability to quickly learn and apply. They say you're able to be a good student.

For the bar exam or USMLE you don't want to identify who is able to learn. You need to see who actually did the learning. Whether it took them 100 hours or 10,000 hours, your #1 priority is to make sure that people can't pass it without having learned the knowledge they need to practice.

Also isn't the proper analogy the actual boards taken in residency and maybe Step 3? Because graduating from med school alone doesn't grant practice rights as opposed to just passing a bar exam to become a lawyer.