Correlation between Quality of teaching and STEP scores?

[little_giant]

Deciding which school to go to, and one aspect people seem to make decisions on is the average STEP 1 or STEP 2 CK score of that school.

I see some highly reputable schools with STEP averages in the low 230s or high 220s while "mid-tier" or "low/mid-tier" schools have averages in the mid to high 230s. This made me wonder if I go to school A with an avg score of 230 does that mean I will receive a worse education (or clinical education for step 2) than school B with an avg score of 240?

What correlates with average STEP scores in terms of the medical school? (not talking about MCAT or GPA that are more pre-med school related)

---

Members don't see this ad.

 

[OnePunchBiopsy]

First off, Step 1 will be P/F by the time you take it.

Second, "Quality of teaching" is a metric that is very subjective and hard to objectively measure.

STEP score correlations are only really reliable when evaluated at the individual level. Averages can vary significantly by class, and even more by school. There are far too many confounding variables (school mission, student specialty interests, class STEP anxiety, individual test anxiety, intelligence of the class as a whole) to compare curriculum quality to STEP scores to generate a meaningful conclusion.

The only metric I've seen positively correlated with a higher STEP 1 score is the number of Qbank questions completed and reviewed. By that measure, institution paid access to Uworld for all students would mean a higher quality curriculum (obviously not true, just hyperbole).

Lastly, if it's worth anything, my school did a great job providing diverse ways of learning lecture content: video recordings, non-mandatory attendance, some PBL (not ALL PBL), etc. Be weary of schools that have mandatory attendance for all lectures and don't record lectures; this will get old. Traditional didactics loss audience attention after 10 minutes, and the audience retains only 20% of a traditional didactic's content.

Hope this helps.

 

[little_giant]

First off, Step 1 will be P/F by the time you take it.

Second, "Quality of teaching" is a metric that is very subjective and hard to objectively measure.

STEP score correlations are only really reliable when evaluated at the individual level. Averages can vary significantly by class, and even more by school. There are far too many confounding variables (school mission, student specialty interests, class STEP anxiety, individual test anxiety, intelligence of the class as a whole) to compare curriculum quality to STEP scores to generate a meaningful conclusion.

The only metric I've seen positively correlated with a higher STEP 1 score is the number of Qbank questions completed and reviewed. By that measure, institution paid access to Uworld for all students would mean a higher quality curriculum (obviously not true, just hyperbole).

Lastly, if it's worth anything, my school did a great job providing diverse ways of learning lecture content: video recordings, non-mandatory attendance, some PBL (not ALL PBL), etc. Be weary of school that have mandatory attendance for all lectures and don't record lectures, this will get old. Traditional didactics loss audience attention after 10 minutes, and the audience retains only 20% of a traditional didactic's content.

Hope this helps.

Thank you so much! Regarding the school mission variable, so one school is a state school and I understand that a big part of the mission would be to produce many primary care physicians, and specialities in primary care in general don't have extremely high STEP scores.

But I dont understand why a school would teach or do something differently that would result in a lower STEP average even if they had this mission.

 

[OnePunchBiopsy]

Thank you so much! Regarding the school mission variable, so one school is a state school and I understand that a big part of the mission would be to produce many primary care physicians, and specialities in primary care in general don't have extremely high STEP scores.

But I dont understand why a school would teach or do something differently that would result in a lower STEP average even if they had this mission.

Not all state schools aim to produce primary care physicians. Check your school's aims and mission statement just to be sure. It should be on their website.

School's are not out to lower scores and force students into primary care. I'm sure every school genuinely wants their students to succeed. Usually, persuading students into primary care is done through early access to clinical experiences in primary care, not a bad curriculum meant to lower Step scores.

Think about it, major academic centers still train primary care specialties, and schools would promote the fact they can match students into top residencies.

 

[Moko]

This made me wonder if I go to school A with an avg score of 230 does that mean I will receive a worse education (or clinical education for step 2) than school B with an avg score of 240? ... But I dont understand why a school would teach or do something differently that would result in a lower STEP average even if they had this mission.

Step scores are not correlated with the quality of clinical education. There would be major deficiencies in (and disservices to) a student's training if their school did nothing else but strictly taught to the boards and focused on testing strategies. An individual's Step score has more to do with the amount of time dedicated to board studying, number of questions completed, and their aptitude for test taking.

It's also important to emphasize that a higher Step score does not mean that a student will become a better resident, physician, and/or colleague (just as how a higher MCAT does not mean that someone will become a better medical student or future physician). Just my thoughts

 

[little_giant]

Step scores are not correlated with the quality of clinical education. There would be major deficiencies in (and disservices to) a student's training if their school did nothing else but strictly taught to the boards and focused on testing strategies. An individual's Step score has more to do with the amount of time dedicated to board studying, number of questions completed, and their aptitude for test taking.

It's also important to emphasize that a higher Step score does not mean that a student will become a better resident, physician, and/or colleague (just as how a higher MCAT does not mean that someone will become a better medical student or future physician). Just my thoughts

Thank you! So regardless of school, average scores are more a representation of the student's own determination and not the school's professors or courses

 

[Goro]

Deciding which school to go to, and one aspect people seem to make decisions on is the average STEP 1 or STEP 2 CK score of that school.

I see some highly reputable schools with STEP averages in the low 230s or high 220s while "mid-tier" or "low/mid-tier" schools have averages in the mid to high 230s. This made me wonder if I go to school A with an avg score of 230 does that mean I will receive a worse education (or clinical education for step 2) than school B with an avg score of 240?

What correlates with average STEP scores in terms of the medical school? (not talking about MCAT or GPA that are more pre-med school related)

I firmly believe that Step 1/Level I scores are more are reflection of the students themselves, rather than the quality of teaching. Most students self-learn nowadays. As an example, two of our worst faculty members (in terms of student evals) yield average or > avg average discipline scores on student's COMLEX performance.

Clinical education can affect our Level II scores. This pattern we learned the hard way was when preceptors didn't have good oversight. Now we have a better leash on them.

 

[efle]

Step 1 scores are reflective of the student body's emphasis on Step 1, not a reflection of professors or individuals.

Take the recent trend in Step 1 scores for Johns Hopkins students as an example (national in green, Hopkins in blue).

234 --> 245 --> 248. Roughly from average to 250 median in just a couple years.

The Genes to Society curriculum did not change. The professors did not change. The student body average GPA and MCAT did not change. The only thing that changed was the student body collectively realized Step 1 was too important to have such mediocre scores and a new zanki/UFAPS Zeitgest swept through the students.

As someone noted above, Step 1 was being used so inappropriately the the authors have decided not to allow students or PDs to know their score any more, so this is all moot for you. But in case Step 2 CK takes over in a similar role before they strike it down too, it's worth showing that it has nothing to do with curriculum/professors and everything to do with how much people prioritize UFAPS.

 

[little_giant]

Step 1 scores are reflective of the student body's emphasis on Step 1, not a reflection of professors or individuals.

Take the recent trend in Step 1 scores for Johns Hopkins students as an example (national in green, Hopkins in blue).
234 --> 245 --> 248. Roughly from average to 250 median in just a couple years.

The Genes to Society curriculum did not change. The professors did not change. The student body average GPA and MCAT did not change. The only thing that changed was the student body collectively realized Step 1 was too important to have such mediocre scores and a new zanki/UFAPS Zeitgest swept through the students.

As someone noted above, Step 1 was being used so inappropriately the the authors have decided not to allow students or PDs to know their score any more, so this is all moot for you. But in case Step 2 CK takes over in a similar role before they strike it down too, it's worth showing that it has nothing to do with curriculum/professors and everything to do with how much people prioritize UFAPS.

Thanks for this! Though I have some doubts that Hopkins students suddenly realized the importance of STEP 1 only ~4 years ago when STEP 1 was a huge factor way before this. I wonder if there is another explanation for this?

 

[efle]

Thanks for this! Though I have some doubts that Hopkins students suddenly realized the importance of STEP 1 only ~4 years ago when STEP 1 was a huge factor way before this. I wonder if there is another explanation for this?

I lived it, as far as I can see it was a Zeitgeist thing. The rise of UFAPS flashcarding was meteoric, from hardly anyone doing it in years above me to nearly everyone doing it in years below me. While step 1 was important prior to 2016, the advising coming down was always that step 1 was a small part of the app, Hopkins name will carry you, learn for the wards not the boards, etc. That bubble finally burst and reality came knocking, and Hopkins scores suddenly caught up by jumping 10+ points to equal HMS, Penn & WashU in a single cohort.

 

[little_giant]

I lived it, as far as I can see it was a Zeitgeist thing. The rise of UFAPS flashcarding was meteoric, from hardly anyone doing it in years above me to nearly everyone doing it in years below me. While step 1 was important prior to 2016, the advising coming down was always that step 1 was a small part of the app, Hopkins name will carry you, learn for the wards not the boards, etc. That bubble finally burst and reality came knocking, and Hopkins scores suddenly caught up by jumping 10+ points to equal HMS, Penn & WashU in a single cohort.

Oh wow that makes sense. What about STEP 2 ck though? Since there isn't really an equivalent to UFAPS for STEP 2. Is there similar data or experiences that show it's just how much students prioritize the exam/how much studying rather than quality of clinical education that result in differences in average scores? Or as Goro said, does quality of clinical education become more of a factor?

 

[efle]

Oh wow that makes sense. What about STEP 2 ck though? Since there isn't really an equivalent to UFAPS for STEP 2. Is there similar data or experiences that show it's just how much students prioritize the exam/how much studying rather than quality of clinical education that result in differences in average scores? Or as Goro said, does quality of clinical education become more of a factor?

Step 2 is an entirely different animal that few people gave much thought to. For the vast majority, ERAS is submitted with only Step 1 visible, and so that's what determines your interview invites. Step 2 can be added later, which is advised to people with weak Step 1 scores they need to compensate for, but for people with strong Step 1 scores it's actually advised to take Step 2 late so it is absent. Many places don't even require Step 2 for ranking - you can literally get your Match without the PDs ever having access to your Step 2 CK score at all.

The prep for it is also very different. Step 1 has the universal UFAPS/anki resources that everyone grinds through in preclinical and then ~2 months of dedicated prep. Step 2 usually gets a couple weeks of reviewing miscellaneous crap with no similar standard curriculum.

Things may rapidly evolve now that step 1 is going pass fail. But for the current generation of students, Step 2 CK is an afterthought compared to Step 1 that has no real impact for the typical person's residency match. Wouldn't give any attention to Step 2 average scores at all.

 

[aldol16]

Why do you care about Step 1 when your Step 1 score will be P/F? It literally will not matter how well you do as long as you're above the passing threshold.

Regarding Step 2, it's even harder to predict. It's an exam that's multiple years out and the Step exams typically have more correlation with how hard you work during med school and studying for the exam than it does with quality of instruction. IMO.

 

[little_giant]

Why do you care about Step 1 when your Step 1 score will be P/F? It literally will not matter how well you do as long as you're above the passing threshold.

Regarding Step 2, it's even harder to predict. It's an exam that's multiple years out and the Step exams typically have more correlation with how hard you work during med school and studying for the exam than it does with quality of instruction. IMO.

Even if STEP 1 becomes p/f for my year, I figured that if there indeed was a correlation between STEP 1 scores and quality of teaching, it would help me choose a school that had a better pre-clinical education.

Now that it seems like the consensus is that the student's effort is the most important factor, I know these numbers aren't really that important... but it's still interesting on an individual scale to see some good schools with low average STEP 1 scores.

 

[efle]

Even if STEP 1 becomes p/f for my year, I figured that if there indeed was a correlation between STEP 1 scores and quality of teaching, it would help me choose a school that had a better pre-clinical education.

Now that it seems like the consensus is that the student's effort is the most important factor, I know these numbers aren't really that important... but it's still interesting on an individual scale to see some good schools with low average STEP 1 scores.

Yeah, when you see tables like this one, I promise it isn't because the professors at UCSF or Hopkins suck compared to the ones at Penn and WashU. Everyone uses the same UFAPS curriculum anyways, nobody is using old professor's slide decks during dedicated. It's just a question of how hard and how soon you start hitting the boards materials.

Or at least, it used to be. The Pass/Fail change might have put that to bed

 

[little_giant]

Yeah, when you see tables like this one, I promise it isn't because the professors at UCSF or Hopkins suck compared to the ones at Penn and WashU. Everyone uses the same UFAPS curriculum anyways, nobody is using old professor's slide decks during dedicated. It's just a question of how hard and how soon you start hitting the boards materials.

Or at least, it used to be. The Pass/Fail change might have put that to bed

Thinking about it now, I think it's for the best that it became P/F.

STEP 2 being the new STEP 1 is probably good because it supposedly tests on more relevant clinical stuff and also the clinical education will become more worthy/valuable of the hefty price tag we pay.

 

[altblue]

Thinking about it now, I think it's for the best that it became P/F.

STEP 2 being the new STEP 1 is probably good because it supposedly tests on more relevant clinical stuff and also the clinical education will become more worthy/valuable of the hefty price tag we pay.

It's for the best until you realize that every Ortho and Derm program in the United States will be filled with Harvard and Penn grads

Step 2 won't be used to discriminate between applicants, its timeline is far too variable

Instead schools will look at prestige

This doesn't affect me because I have negative interest in surgery/derm and more importantly our class will still have scored step, but for capable people at lesser known schools it will def become an issue

 

[little_giant]

It's for the best until you realize that every Ortho and Derm program in the United States will be filled with Harvard and Penn grads

Step 2 won't be used to discriminate between applicants, its timeline is far too variable

Instead schools will look at prestige

This doesn't affect me because I have negative interest in surgery/derm and more importantly our class will still have step, but for capable people at lesser known schools it will def become an issue

I thought everyone agreed that prestige would only become a bigger issue for low tier/DO/IMGs?

 

[treatem-and-streetem]

I thought everyone agreed that prestige would only become a bigger issue for low tier/DO/IMGs?

Nope. The MCAT got that much more important now. I go to mid tier MD school and we all know that this change made getting into a more prestigious school even more valuable.

 

[altblue]

I thought everyone agreed that prestige would only become a bigger issue for low tier/DO/IMGs?

it'll probably scale with the prestige of the school, top 5 > top 15 > top 20 etc.

I do agree that community-based MD programs without home programs in subspecialty surgery/derm/rad onc, DOs and IMGs will be hit the hardest though

 

[efle]

Not to derail this into another pass/fail Step 1 debate, but OP, I wouldn't treat prestige as the new be-all end-all like you'll hear on forums. 15+ years ago when Step 1 scores were ignored (or didn't even exist yet) the mid-tier and low-tier medical schools still sent their share of students into surgical fields.

From Charting Outcomes, the percentage of matched applicants coming from Top 40 NIH Med Schools:

All Specialties: 32%
Ortho: 32% (no difference!)
ENT: 30% (lower!)
Plastic: 40%
Neurosurg: 44%

So, there's the data, depending on the surgical subspecialty there's either no skew or a tiny skew of ~1/10th of spots being over-represented

 

[altblue]

Not to derail this into another pass/fail Step 1 debate, but OP, I wouldn't treat prestige as the new be-all end-all like you'll hear on forums. 15+ years ago when Step 1 scores were ignored (or didn't even exist yet) the mid-tier and low-tier medical schools still sent their share of students into surgical fields.

From Charting Outcomes, the percentage of matched applicants coming from Top 40 NIH Med Schools:

All Specialties: 32%
Ortho: 32% (no difference!)
ENT: 30% (lower!)
Plastic: 40%
Neurosurg: 44%

So, there's the data, depending on the surgical subspecialty there's either no skew or a tiny skew of ~1/10th of spots being over-represented

That's a great point and it's def possible pass/fail step 1 doesn't disadvantage people from non-well known programs, but it doesn't necessarily mean applications go back to being evaluated that way. The times have changed and with more prominent college/med school rankings prestige is more visible in our culture... and who's to say people won't find another easy, obvious way to discriminate between lots of applications after relying on another method, Step1, for so long

On some level I don't want to derail either, but I feel this conversation is still relevant because the impact of new step on possible school choice

 

[longhaul3]

It's for the best until you realize that every Ortho and Derm program in the United States will be filled with Harvard and Penn grads

Step 2 won't be used to discriminate between applicants, its timeline is far too variable

Instead schools will look at prestige

The timeline is only variable because the score doesn't matter. I think you will see a rapid change once there are no more Step 1 scores. Everyone will take CK early. It may not become officially standardized, but I would imagine a lot of programs will officially or unofficially indicate that they will require a CK score for an interview. There will also be a lot of self and peer pressure (i.e. everyone else is doing it) to take it early to have something to lean on for ERAS.

 

[aldol16]

Thinking about it now, I think it's for the best that it became P/F.

STEP 2 being the new STEP 1 is probably good because it supposedly tests on more relevant clinical stuff and also the clinical education will become more worthy/valuable of the hefty price tag we pay.

That's a very superficial view. It's a terrible idea to have Step 2 being the new Step 1 because of the timing of Step 2. Since Step 2 is predominantly a clinical exam, it will always be after the clinical year of clerkships (or very near the end of the clinical year). That means that you have a matter of months between getting your Step 2 score (or in many cases, you won't have your score) before ERAS is due. That means that you could have spent your entire first three years preparing for a career in Derm but all of a sudden your Step score is too low to be competitive for that. Now what? Take a research year to figure your life out? Apply into medicine?

 

[efle]

That's a very superficial view. It's a terrible idea to have Step 2 being the new Step 1 because of the timing of Step 2. Since Step 2 is predominantly a clinical exam, it will always be after the clinical year of clerkships (or very near the end of the clinical year). That means that you have a matter of months between getting your Step 2 score (or in many cases, you won't have your score) before ERAS is due. That means that you could have spent your entire first three years preparing for a career in Derm but all of a sudden your Step score is too low to be competitive for that. Now what? Take a research year to figure your life out? Apply into medicine?

Many schools already have students taking Step 1 after MS3/core rotations...this hasn't been an issue for that model.

 

[aldol16]

Many schools already have students taking Step 1 after MS3/core rotations...this hasn't been an issue for that model.

A lot of those schools have pushed their curriculum forward by moving to 1.5 or 1 year curricula. If you take Step 1 after MS3 year, then you're also basically pigeon-holing yourself. Unless you really think that people can in any way be relatively assured of a particular Step 1 score.

 

[efle]

A lot of those schools have pushed their curriculum forward by moving to 1.5 or 1 year curricula. If you take Step 1 after MS3 year, then you're also basically pigeon-holing yourself. Unless you really think that people can in any way be relatively assured of a particular Step 1 score.

Fair, and considering most of the schools that launched this style have Step 1 averages in the 240s, it's probably much safer move for them to take it late.

I think Step 2 CK is ultimately destined for the same fate as Step 1 if students and PDs start to treat it as the new substitute. What we need is application caps, but that's for another thread.

 

[aldol16]

I think Step 2 CK is ultimately destined for the same fate as Step 1 if students and PDs start to treat it as the new substitute. What we need is application caps, but that's for another thread.

I'm honestly surprised they didn't think about this when they made Step 1 P/F. Or rather, they probably thought of it then punted. It's the natural next step to make Step 2 P/F, which would then turn the residency application process into a free-for-all dominated by students from the top schools.