We all know clerkship grading is highly variable, but I hadn't seen it quantified before:

This forum made possible through the generous support of SDN members, donors, and sponsors. Thank you.

efle

not an elf
10+ Year Member
Joined
Apr 6, 2014
Messages
14,146
Reaction score
22,788
I'm sure we all know from experience that clerkship grading is subjective and variable. But, I didn't realize the degree of variation that existed between schools, or between departments within one school.

Some schools are out here giving less than 10% Honors while others award it to >80%!

Even within an individual school, you can have one clerkship giving 72% Honors in Medicine while only giving 28% Honors in Peds. Yikes, better hope you prefer to treat adults.





I know the MSPEs can contain grade distributions to give PDs some context. But do we really think they've got the time to reference the histogram for all the grades on a transcript? No way, it's all about glance value.

No real point of this post other than to vent, I suppose. Nice to know we aren't collectively crazy, the data really does show the MS3 grading experience is just as arbitrary and variable as it feels.

Source pubmed ID - 22722356

Members don't see this ad.
 
My school was like this. Didn't matter what your evals said or your shelf score. Only 10% of the class got honors. I always felt it was a way for the admin to pick and choose who gets a ton of honors and who gets only a few and in which fields. Even if you aced the rotation, it's easy for them to tell you that 10% aced it better than you did.

I'm all in favor of transparency. Have a pre-decided range of shelf scores and a criteria for evals that equal honors. If only 10$ of the class hits it, only 10% gets honors, but if 80% of the class hits it, then 80% gets honors.
 
Members don't see this ad :)
So glad to not be in the P/F step 1 era.

3uk34f.jpg
 
They know which schools give honors out like candy after doing it year after year. Also the histogram is literally right next to the grade
 
They know which schools give honors out like candy after doing it year after year. Also the histogram is literally right next to the grade
But it seems to only be specific departments within each school that do so. Do they really remember each school's grading in each area?

I thought the transcript was separate from the MSPE, with the former being a summary for glancing at and the latter being much more thorough with all the written comments, histogram etc
 
My take is now that Step is going pass/fail, something is going to have to give in regards to stratifying applicants. Either schools will make preclinical years a graded system or they need to be 1000% transparent about clinical grading because people's careers are at an even higher stake than before, particularly those going for very competitive specialties.
 
My take is now that Step is going pass/fail, something is going to have to give in regards to stratifying applicants. Either schools will make preclinical years a graded system or they need to be 1000% transparent about clinical grading because people's careers are at an even higher stake than before, particularly those going for very competitive specialties.
20+ years ago a handful of elite medical schools rolled out the Pass/Fail preclinical model. It was popular and spread rapidly, and these days every single one of the Top 20 med schools has Pass/Fail preclinical.

In recent years, some of these same elite handful (like Harvard and Vanderbilt) have started rolling out Pass/Fail clerkship grading. Their match lists still seem fantastic. Wouldn't surprise me at all if this becomes the next popular trend.

The net effect is that in the near future, at certain schools, an ERAS will show only Pass/Fail preclinical, Pass/Fail Step 1, and Pass/Fail clerkships. Good luck to those PDs.

Over in the Step 1 Pass/Fail megathread, one of the interesting ideas that got floated was to have all medical schools use the standardized NBME shelves for preclinical and report those NBME scores in the MSPE. This would be like comparing applicants by their Step 1, but instead of trying to measure their abilities with a single day of testing and a wide error interval, they could look at your overall performance across 1000+ test questions and across months and years.

Seems like a decent idea to improve on the Step 1 system, thought I'm sure it would make preclinical even more stressful and boards-focused.
 
20+ years ago a handful of elite medical schools rolled out the Pass/Fail preclinical model. It was popular and spread rapidly, and these days every single one of the Top 20 med schools has Pass/Fail preclinical.

In recent years, some of these same elite handful (like Harvard and Vanderbilt) have started rolling out Pass/Fail clerkship grading. Their match lists still seem fantastic. Wouldn't surprise me at all if this becomes the next popular trend.

The net effect is that in the near future, at certain schools, an ERAS will show only Pass/Fail preclinical, Pass/Fail Step 1, and Pass/Fail clerkships. Good luck to those PDs.

Over in the Step 1 Pass/Fail megathread, one of the interesting ideas that got floated was to have all medical schools use the standardized NBME shelves for preclinical and report those NBME scores in the MSPE. This would be like comparing applicants by their Step 1, but instead of trying to measure their abilities with a single day of testing and a wide error interval, they could look at your overall performance across 1000+ test questions and across months and years.

Seems like a decent idea to improve on the Step 1 system, thought I'm sure it would make preclinical even more stressful and boards-focused.

It then becomes where you went to school and LORs which will cause even more self-selection and those match lists for the top 20 programs will be even more crazy. Why do it for preclinicals though instead of doing it for 3rd year?
 
I think my school handled clerkship grading about as well as it can be handled. They used a true honors system where only about 20% of the class in each clerkship can get honors, weighing the shelf, evals, and other smaller assignments. At the end of the year if only 10% or so got honors, they would adjust the thresholds accordingly. It usually worked out so that a good shelf score was necessary but not sufficient for honors but the one inevitable straight 2/4 eval wouldn’t hamstring you. Obviously there’s always going to be an element of subjectivity to clerkship grading but I do think it’s important to have some form of standardization. I also think it’s important to give people a chance to really shine and prove that they earned honors - I’ve heard of some schools that give “Pass” only to the absolute bottom percentile and give everyone else some form of “honors, high honors, etc.” I know on my MSPE for each clerkship there was a big arrow pointing exactly to where I fell along the grade distribution and that seemed to be a good way of making the grading clear.
 
I think my school handled clerkship grading about as well as it can be handled. They used a true honors system where only about 20% of the class in each clerkship can get honors, weighing the shelf, evals, and other smaller assignments. At the end of the year if only 10% or so got honors, they would adjust the thresholds accordingly. It usually worked out so that a good shelf score was necessary but not sufficient for honors but the one inevitable straight 2/4 eval wouldn’t hamstring you. Obviously there’s always going to be an element of subjectivity to clerkship grading but I do think it’s important to have some form of standardization. I also think it’s important to give people a chance to really shine and prove that they earned honors - I’ve heard of some schools that give “Pass” only to the absolute bottom percentile and give everyone else some form of “honors, high honors, etc.” I know on my MSPE for each clerkship there was a big arrow pointing exactly to where I fell along the grade distribution and that seemed to be a good way of making the grading clear.

I like this system a lot. Gives the student some much needed wiggle room if something unfortunate happens.
 
My school is extremely variable between sites. Some sites have only residents evaluate you (LCME violation), some have only attendings evaluate you(another LCME violation), and some have both (LCME rule).

I’ve seen the internal stats on grading between sites and some sites give almost exclusively honors while other give only 10% honors. Our grades are basically exclusively dependent upon which site you randomly get assigned.
 
The other unfortunate factor is that a lot of attendings/residents don't realize that if they give us 3/5 (which is denoted to mean "performance at level of current training") it actually translates to a 60% after being converted to an overall grade. Some do realize this and therefore strive to give higher numbers but inevitably people end up screwed over because of this despite performing similarly to their peers.
 
Members don't see this ad :)
I feel bad for future med students. There was definitely a feeling of relief when step 1 was over with, now you’re going to feel pressured and absolutely have no idea where you stand in competitiveness until step 2 is done, presumably right before you apply. Imagine gearing yourself up for a sub specialty the whole time then don’t do as well on step 2 and all your rotations and letters were towards that. Also I feel like there isn’t going to be as much of a spread of scores for step 2 because it’s an easier exam. You’re going to have to get in the 270’s to differentiate yourself
 
I feel bad for future med students. There was definitely a feeling of relief when step 1 was over with, now you’re going to feel pressured and absolutely have no idea where you stand in competitiveness until step 2 is done, presumably right before you apply. Imagine gearing yourself up for a sub specialty the whole time then don’t do as well on step 2 and all your rotations and letters were towards that. Also I feel like there isn’t going to be as much of a spread of scores for step 2 because it’s an easier exam. You’re going to have to get in the 270’s to differentiate yourself

Well at least the top schools are happy as they also push strongly for P/F clinical years.
 
Well at least the top schools are happy as they also push strongly for P/F clinical years.
Yea the students at top schools always had an advantage enough where the scores mattered less. But if you’re coming from a not so top school all those grades were essential for going to a good residency program. Now your med school will matter a lot more and your individual work ethic and motivation won’t. Sucks but that’s just the way it goes!
 
Yea the students at top schools always had an advantage enough where the scores mattered less. But if you’re coming from a not so top school all those grades were essential for going to a good residency program. Now your med school will matter a lot more and your individual work ethic and motivation won’t. Sucks but that’s just the way it goes!

I'm just hoping specialties and academic programs don't lock anyone out because of school name.
 
My school was like this. Didn't matter what your evals said or your shelf score. Only 10% of the class got honors. I always felt it was a way for the admin to pick and choose who gets a ton of honors and who gets only a few and in which fields. Even if you aced the rotation, it's easy for them to tell you that 10% aced it better than you did.

I'm all in favor of transparency. Have a pre-decided range of shelf scores and a criteria for evals that equal honors. If only 10$ of the class hits it, only 10% gets honors, but if 80% of the class hits it, then 80% gets honors.

Many schools do this already, but it should be instituted nationwide.
 
Or, at the very least, there should be a standard set within each institution. A blanket policy of X% Honors, Y% High Pass, Z% Pass across all clerkships would let you read a transcript at a glance
Honestly I feel like a lot of people should get honors and a lot pass and a small percentage fail if they’re a sociopath or something.

A lot of people will think someone is worse just because they don’t have a personality similar to their own and they misconstrue that to mean the person isn’t good. I just honor everybody because the system sucks and just because I wouldn’t hang out with you after work it doesn’t mean I should stand in the way of their specialty choice. 3rd year goal should be to figure out what you wanna dowith your life, and objective stuff should drive the rest. Tests aren’t perfect but at least it’s objective.
 
Honestly I feel like a lot of people should get honors and a lot pass and a small percentage fail if they’re a sociopath or something.

A lot of people will think someone is worse just because they don’t have a personality similar to their own and they misconstrue that to mean the person isn’t good. I just honor everybody because the system sucks and just because I wouldn’t hang out with you after work it doesn’t mean I should stand in the way of their specialty choice. 3rd year goal should be to figure out what you wanna dowith your life, and objective stuff should drive the rest. Tests aren’t perfect but at least it’s objective.
I will also be pan-Honoring any medical student I ever work with. Unless you actively disrespect people or interfere with work, it's 5/5s for you and dismissal at noon.
 
Man, im so glad that step 1 is pass/fail now. I think the next step is to make shelf scores pass/fail so we can remove all objectivity from our lives. Im sure the error bars on the IM shelf is wide enough that all the people getting those 10% honors had a portion of their bar in the HP territory.
 
Man, im so glad that step 1 is pass/fail now. I think the next step is to make shelf scores pass/fail so we can remove all objectivity from our lives. Im sure the error bars on the IM shelf is wide enough that all the people getting those 10% honors had a portion of their bar in the HP territory.
It's the year 2030. All residency spots are secured by impressing on away rotations and the letters they generate. Students who despise this system and want 2020 back are laughed at for being a bunch of flashcard nerds that belong in Rads and Path.

Utopia or dystopia, you decide
 
That's the system in Canada. No standardized exams are used to differentiate candidates, and all medical schools are entirely pass-fail grading for all years. Interview invites are determined by away rotations and letters, with maybe some influence from your CV.
 
That's the system in Canada. No standardized exams are used to differentiate candidates, and all medical schools are entirely pass-fail grading for all years. Interview invites are determined by away rotations and letters, with maybe some influence from your CV.
Is this perceived as a generally fair and good system by Canadian MDs? Or do they wish there was something objective they could use to out-compete peers for the most popular residencies?
 
Interesting question, and the answer probably depends on who you ask. My take:

Many program directors don't like the system because it makes it difficult to objectively differentiate candidates.

A lot of Canadian MDs think performance on the wards is more important than performance on a test, so I would guess that most of them would not support a move to a more American-style system.

Most students do appreciate not having a single high-stakes exam that can permanently close the door on your desired career if you don't do well on it.

They also appreciate the pass-fail nature of medical school, because it dampens the amount of overt competition a bit. However there is still a ton of covert competition and gunning due curved grading making it possible to fail even if you prepared for an exam, and internal ranking that determines scholarships & awards (which are valuable for residency applications). Some are only awarded after the match, but some are awarded privately at the end of years 1, 2, and 3.

Well-off and well-connected students seem to like the system because they are able to use their (parents') financial resources to do away rotations at as many schools as possible, and use their (parents') connections to get glowing letters from influential people. And a large proportion of medical students in Canada come from upper middle class or above backgrounds, and/or have at least one parent who is a doctor. So that is a common view. Most students feel that if you take an interest in a normal specialty and put in the work to get there, it's attainable. However for plastic surgery and dermatology, no amount of work can guarantee you a spot because there are roughly twice as many applicants as spots in those specialties.

There are almost no places for IMGs in the Canadian match. There are also no DO schools.
 
A lot of Canadian MDs think performance on the wards is more important than performance on a test, so I would guess that most of them would not support a move to a more American-style system.

I think everyone can agree to that. I am strongly against a P/F step 1 exam. However, if clerkship grades actually did accurately assess clinical knowledge I would change my position. The problem is that they do not.
 
Interesting question, and the answer probably depends on who you ask. My take:

Many program directors don't like the system because it makes it difficult to objectively differentiate candidates.

A lot of Canadian MDs think performance on the wards is more important than performance on a test, so I would guess that most of them would not support a move to a more American-style system.

Most students do appreciate not having a single high-stakes exam that can permanently close the door on your desired career if you don't do well on it.

They also appreciate the pass-fail nature of medical school, because it dampens the amount of overt competition a bit. However there is still a ton of covert competition and gunning due curved grading making it possible to fail even if you prepared for an exam, and internal ranking that determines scholarships & awards (which are valuable for residency applications). Some are only awarded after the match, but some are awarded privately at the end of years 1, 2, and 3.

Well-off and well-connected students seem to like the system because they are able to use their (parents') financial resources to do away rotations at as many schools as possible, and use their (parents') connections to get glowing letters from influential people. And a large proportion of medical students in Canada come from upper middle class or above backgrounds, and/or have at least one parent who is a doctor. So that is a common view. Most students feel that if you take an interest in a normal specialty and put in the work to get there, it's attainable. However for plastic surgery and dermatology, no amount of work can guarantee you a spot because there are roughly twice as many applicants as spots in those specialties.

There are almost no places for IMGs in the Canadian match. There are also no DO schools.
Fascinating, appreciate the perspective. I wonder if US MDs in future generations, who "grow up" with Step being Pass/Fail, will shift to have a similar view.

Are there significant differences between the Canadian medical schools in terms of their match lists? Like a handful that gobble up all the competitive Derm/Plastics type spots, or do students from any school have an equal shot?
 
I think everyone can agree to that. I am strongly against a P/F step 1 exam. However, if clerkship grades actually did accurately assess clinical knowledge I would change my position. The problem is that they do not.
It's really shocking how little effort schools are willing to put into the validity of their clinical grading.

A simple correction for the average grade awarded by each evaluator would be huge. For example, if Attending A gives an average of 3.0 and Attending B gives an average of 4.5, then a student given a 3/5 by A and a different student given a 4.5/5 by B should both be entered into the gradebook as equivalent.

Yet I have not heard of anything like this being done anywhere ever!
 
I don't think all clinical grading is subjective.
If you look at all the clinical grades for a particular student, you can make out who is excelling and who is average.
What I probably would do is take out the numerical grades (like 3/5,4/5 etc) and only have preceptors write what they thought of the student. I would expect all the preceptors who think they have the ability to analyze a students' performance to write a detailed evaluation, at least 4-5 sentences long with specific examples of what they thought the student did well on and what they could improve on (not if they gelled with them on a personal level but talking about the students attitude, ability to adapt, communication skills, etc)
If a preceptor is writing one sentence evaluation then they probably do not have enough interaction with the student and shouldn't be grading the student in the first place.
Much better of way of looking at the students' clinical performance than arbitrary numbers that different preceptors have different definitions.
 
I don't think all clinical grading is subjective.
If you look at all the clinical grades for a particular student, you can make out who is excelling and who is average.
What I probably would do is take out the numerical grades (like 3/5,4/5 etc) and only have preceptors write what they thought of the student. I would expect all the preceptors who think they have the ability to analyze a students' performance to write a detailed evaluation, at least 4-5 sentences long with specific examples of what they thought the student did well on and what they could improve on (not if they gelled with them on a personal level but talking about the students attitude, ability to adapt, communication skills, etc)
If a preceptor is writing one sentence evaluation then they probably do not have enough interaction with the student and shouldn't be grading the student in the first place.
Much better of way of looking at the students' clinical performance than arbitrary numbers that different preceptors have different definitions.
I dunno dude, I've spent weeks on services with people just to have them put "good work keep reading" type BS in my eval box. Some people, especially overworked residents, just don't have the time or energy or desire to notice the minor differences between the dozens of students they work with each year.
 
It's really shocking how little effort schools are willing to put into the validity of their clinical grading.

A simple correction for the average grade awarded by each evaluator would be huge. For example, if Attending A gives an average of 3.0 and Attending B gives an average of 4.5, then a student given a 3/5 by A and a different student given a 4.5/5 by B should both be entered into the gradebook as equivalent.

Yet I have not heard of anything like this being done anywhere ever!

One medical school to my knowledge utilizes a clinical grading scheme similar to what you've described. They adjust grading based on historical trends by evaluators. If Attending A who historically gives out 3's gives a student a 4 then that student will be held in higher regard than a student who gets a 3.0 and is evaluated by Attending B who historically gives out 4.5's.
 
One medical school to my knowledge utilizes a clinical grading scheme similar to what you've described. They adjust grading based on historical trends by evaluators ("Hawks and Doves"). If Attending A who historically gives out 3's gives a student a 4 then that student will be held in higher regard than a student who gets a 3.0 and is evaluated by Attending B who historically gives out 4.5's.
So it CAN be done. This should be standard practice, otherwise you get nonsense like certain hospital sites being a guarunteed Honors while other sites make it impossible, based on who the attendings are.
 
Are there significant differences between the Canadian medical schools in terms of their match lists? Like a handful that gobble up all the competitive Derm/Plastics type spots, or do students from any school have an equal shot?

I don't think Canadian schools have nearly the same varying levels of "prestige" that US schools do.
 
I don't think Canadian schools have nearly the same varying levels of "prestige" that US schools do.
If I recall correctly they're all absurdly competitive, like requiring a top 10-20% score in every MCAT subsection to even read your application? Probably no concept of reach vs safety in that environment
 
I dunno dude, I've spent weeks on services with people just to have them put "good work keep reading" type BS in my eval box. Some people, especially overworked residents, just don't have the time or energy or desire to notice the minor differences between the dozens of students they work with each year.
I understand your point but the reason we pay 50k for 3rd year is partly so we have supervised preceptorship and written feedback is a huge part of that. When LCME gives medical school accreditation, feedback and supervision is a big part of what they look at.
I don't think residents necessarily have to write a written evaluation for each medical student and rightfully so.
The whole team consisting of the intern, resident and attending should discuss and come up with a final evaluation which the attending should write. Part of an academic attending's job responsibility is supervising medical students and providing feedback.
If the attending doesn't have any feedback to give, then the medical school should know that and corrective measures should be taken place to make sure that the student is having significant clinical experience and interaction with the medical team.
 
I understand your point but the reason we pay 50k for 3rd year is partly so we have supervised preceptorship and written feedback is a huge part of that. When LCME gives medical school accreditation, feedback and supervision is a big part of what they look at.
I don't think residents necessarily have to write a written evaluation for each medical student and rightfully so.
The whole team consisting of the intern, resident and attending should discuss and come up with a final evaluation which the attending should write. Part of an academic attending's job responsibility is supervising medical students and providing feedback.
If the attending doesn't have any feedback to give, then the medical school should know that and corrective measures should be taken place to make sure that the student is having significant clinical experience and interaction with the medical team.

Well I guess no one is doing their job then, lol
 
Well I guess no one is doing their job then, lol

I don't think so. I had probably 60% of attendings do an excellent job of supervising, giving regular feedback and teaching.
The other 40% of attendings: the med school either needs to do a better job of coaching those attendings, placing students with different attendings who do like to teach, or just simply ignoring those evals.
Had friends that worked their asses off on sub-i's and third yr rotations only for attendings to say: "good job, read more".
 
I don't think so. I had probably 60% of attendings do an excellent job of supervising, giving regular feedback and teaching.
The other 40% of attendings: the med school either needs to do a better job of coaching those attendings, placing students with different attendings who do like to teach, or just simply ignoring those evals.
Had friends that worked their asses off on sub-i's and third yr rotations only for attendings to say: "good job, read more".

But you're proving my point. No one is doing their jobs. 40%? That's poor. With the amount of money we're paying, that's total nonsense.
 
Last edited:
I literally got an eval back that was completely blank except for 2 pull down responses, the kind that they have to do in order to submit the eval, and then the solid P on the last page (H/P/F clinical grades). Never even met the attending that was in charge of evals.
 
But you're proving my point. No one is doing their jobs. 40%? That's poor. With the amount of money we're paying, that's total nonsense.
Not disagreeing at all. As I said before, we are paying 50,000 a year and feedback and supervision is a part of that education. Med school is paying deans and other staff to monitor these things amongst many other things. They are failing at doing their job if these things continue to happen year after year.
 
I literally got an eval back that was completely blank except for 2 pull down responses, the kind that they have to do in order to submit the eval, and then the solid P on the last page (H/P/F clinical grades). Never even met the attending that was in charge of evals.
Yeah. Great we made step 1 pass or fail but if you don't change these glaring mistakes in medical education, making step 1 p/f is not really going to improve medical education that much.
 
Not disagreeing at all. As I said before, we are paying 50,000 a year and feedback and supervision is a part of that education. Med school is paying deans and other staff to monitor these things amongst many other things. They are failing at doing their job if these things continue to happen year after year.

Oh, I see. I misread what you said earlier. My mistake.
 
Fascinating, appreciate the perspective. I wonder if US MDs in future generations, who "grow up" with Step being Pass/Fail, will shift to have a similar view.

Are there significant differences between the Canadian medical schools in terms of their match lists? Like a handful that gobble up all the competitive Derm/Plastics type spots, or do students from any school have an equal shot?
I don't think Canadian schools have nearly the same varying levels of "prestige" that US schools do.
If I recall correctly they're all absurdly competitive, like requiring a top 10-20% score in every MCAT subsection to even read your application? Probably no concept of reach vs safety in that environment

Match lists don't differ significantly between schools, except that some may produce a slightly higher proportion of family doctors. This appears to be because more students from those schools rank family as their first choice.

Canadian medical schools don't vary much in terms of prestige. Some are in more/less desirable locations, which can make them more difficult to get into for residency. Oddly this doesn't hold true for medical school admissions, where schools in some of the worst locations have among the lowest admit rates. Some schools market themselves more in the US though, so they have better name recognition. All Canadian medical schools are at academic centers, and all receive a lot of research funding from government and private sources. No community residency programs, only academic.

Every school treats the MCAT differently. At some it's a huge component of your application with very high requirements like minimum of top 5-10% on CARS to avoid being auto-rejected. At others it's worth between a middling amount and almost nothing. And I think there are some that don't require or look at the MCAT at all. There are no safety schools. But there are some schools where it is way easier to get into medical school due to reserved seats for in-province students and very few in-province applicants. And there are some schools where it is effectively impossible to get in if you are not from (ie completed high school there and can prove you lived a substantial part of your life there) the specific small geographic region that they explicitly favor for admissions.

For residency, every school consistently takes a very large number of their own graduates, probably due to programs being more familiar with students from their own school. So if you like your medical school/location, there is a pretty good chance you could stay there for residency if you want. The downside of this is a high degree of academic inbreeding in Canadian faculties of medicine. Many Canadian physicians have only ever known one institution of higher learning, from undergrad through medical school and then residency. (Most Canadian undergraduates study at the university closest to their home, since there's not much prestige incentive to travel elsewhere in the country. Same with medical school, and it's generally easiest to get into medical school in your home province).
 
It's really shocking how little effort schools are willing to put into the validity of their clinical grading.

A simple correction for the average grade awarded by each evaluator would be huge. For example, if Attending A gives an average of 3.0 and Attending B gives an average of 4.5, then a student given a 3/5 by A and a different student given a 4.5/5 by B should both be entered into the gradebook as equivalent.

Yet I have not heard of anything like this being done anywhere ever!

My school weights grades. Resident grades mean more than intern or attending grades, and the faculty who run the clerkship normalize the grades so that people aren’t getting honors because they got an easy grader and vice versa.
 
Top