Are surgical residents and attendings good at assessing your clinical knowledge?

This forum made possible through the generous support of SDN members, donors, and sponsors. Thank you.

SLUser11

CRS
15+ Year Member
Joined
Feb 22, 2005
Messages
2,876
Reaction score
814
Please read the attached PDF and let me know what you think.

Are attendings and residents bad at assessing clinical knowledge, or is the shelf bad at assessing clinical knowledge? I've certainly heard complaints about the shelf being unfair, and evaluations being unfair....which one is the better instrument?

(I think I know how this will be answered, but I'm interested in the response nonetheless)

Members don't see this ad.
 

Attachments

  • Am J Surg evals vs shelf scores for clinical knowledge.pdf
    207.5 KB · Views: 100
Please read the attached PDF and let me know what you think.

Are attendings and residents bad at assessing clinical knowledge, or is the shelf bad at assessing clinical knowledge? I've certainly heard complaints about the shelf being unfair, and evaluations being unfair....which one is the better instrument?

(I think I know how this will be answered, but I'm interested in the response nonetheless)

I think the best test of medical student knowledge is the med student seeing a consult, presenting the H&P, and (most particularly) his answer when the resident/attending asks "What do you think is going on and what should we do next?"

Unfortunately, that's a pretty time-intensive way of eval'ing students. I never presented a patient to most of my attendings. Residents are on a case-by-case basis. E.g., I saw plenty of consults on my own on my gen surg rotation, but never on my ENT SubIs.

I think evals should be limited to residents and attendings who had some meaningful interaction with the student.

To speak to the article, it notes that residents/attendings as a group rate medical students pretty reliably. However, these ratings poorly correlate with shelf scores. I think this is for a few reasons:

1) Residents/attendings grade a number of things that the shelf can't possibly test: likability, being a team player, professionalism, clinical skills, attitude, etc. Even though the retrospective study just looked at assessment of student knowledge, I think these other factors definitely color how a student is graded.
2) There's a difference between a) being able to assess a patient and display reasonable clinical judgment and b) being able to play buzzword bingo on shelf exams.
 
Last edited:
  • Like
Reactions: 1 user
Well I think this study is confounding the grade on the evaluation with actual clinical knowledge. I really don't think attendings are thinking deeply about whether or not the student was knowledgeable - just how much they liked them overall. AKA, clinical evals are largely about kissing ass and personality, not clinical knowledge. Even on the section of the eval labeled "clinical knowledge".

And how can a Shelf exam be unfair? You know what's being tested and how it's tested from day one.....
 
Members don't see this ad :)
I remember people not knowing how much detail was on the shelf, or the minimum knowledge necessary to pass. Basically, like for IM, wondering if SUTM was too much knowledge or not...and if there is a smaller resource
 
I remember people not knowing how much detail was on the shelf, or the minimum knowledge necessary to pass. Basically, like for IM, wondering if SUTM was too much knowledge or not...and if there is a smaller resource

Is that the unfair part?
 
I don't think residents and attendings are good at assessing knowledge but are needed for assessing other qualities important in a student. Glaring deficiencies are easier to pick up, but the subtler distinction between students is hit or miss. Not that the shelf is awesome but it is more consistent.
 
Is that the unfair part?

Yea, at least, that was the reasoning for the tests not being fair compared to an in house exam with "all the answers given" or a study guide with everything to know instead of reading some book. I personally preferred a shelf to a poorly written, highly minutiae filled test...
 
Yea, at least, that was the reasoning for the tests not being fair compared to an in house exam with "all the answers given" or a study guide with everything to know instead of reading some book. I personally preferred a shelf to a poorly written, highly minutiae filled test...

I don't know if you meant it that way, but the way your phrased that post it sounded like people thought it was unfair that they couldn't figure out the minimum effort they'd need to skate by and pass. It's not like SUTM was that big....

I think the shelf is the only objective part of 3rd year.
 
  • Like
Reactions: 1 user
I don't know if you meant it that way, but the way your phrased that post it sounded like people thought it was unfair that they couldn't figure out the minimum effort they'd need to skate by and pass. It's not like SUTM was that big....

I think the shelf is the only objective part of 3rd year.

Apparently, people thought I was crazy for managing to get 1 entire pass through that book. Then again, I did force myself to read it all so that I would feel comfortable. It also wasn't small enough to fit in my white coat to carry around unlike Case Files :(
 
I don't know if you meant it that way, but the way your phrased that post it sounded like people thought it was unfair that they couldn't figure out the minimum effort they'd need to skate by and pass. It's not like SUTM was that big....

I think the shelf is the only objective part of 3rd year.

The shelf is objective, sure.

But that doesn't necessarily mean that it is relevant.
 
  • Like
Reactions: 2 users
I would opine that clinical evaluations and shelf scores measure two different skill sets that are both necessary to be a well-rounded doctor.

If anything, the inter-rater reliability between residents and attendings is encouraging, because it shows there's some objectivity to the process. While still imperfect, specifically looking at the "clinical knowledge" portion of the evaluation also removes a portion of the popularity contest.
 
  • Like
Reactions: 1 users
Or we're missing a huge part of the equation here...that attendings often base their evaluations from asking the residents about you (especially if you're in a rotation with minimal attending contact). If the resident didn't like you, you can bet they're going to **** all over you when the attending asks them how you did.

That is a pretty significant confounder - how much of the attendings' evaluation is actually just a mirror of the residents' evaluation. The opinions of the students don't form in a vacuum, and generally the entire team shares the same opinion of the students - both good and bad.
 
  • Like
Reactions: 1 user
I would opine that clinical evaluations and shelf scores measure two different skill sets that are both necessary to be a well-rounded doctor.

My experience was that medical school evaluations were very poor measurements of anything. An evaluation in a workplace is supposed to be a measurement of how well you did your job. In residency, for example, I have generally found my evaluations are a reflection of how rarely the attending needed to correct my management, how I impacted work flow, and how many complaints/compliments I received from the patients and ancillary staff. Unfortunately a medical student in our litigious society very rarely has a job to do. There is nothing objective to judge medical students based on, so they are often judged based on a combination of the student's likability and the resident's mood that day. Those evals felt pretty random and pretty useless.
 
  • Like
Reactions: 4 users
Members don't see this ad :)
Those evals felt pretty random and pretty useless.
I felt the same way about the shelf exam. I've never seen a better disconnect between what I learned in clinic and what I was tested. Though I agree that some evals were ridiculous. I've had my fair share.
 
Lol evals are a total joke. The differences between attendings and let alone instutions makes comparing residency applicants impossible.

They should publish a paper correlating how big your tits and ass are relative to your evaluations. I suspect a strong R value.
 
Shelf exams separate the men from the boiz. Glad they keep the required percentile at 88 for my school so my honors mean something. Gaming my eval portion is cake.
 
Shelf exams separate the men from the boiz. Glad they keep the required percentile at 88 for my school so my honors mean something. Gaming my eval portion is cake.

Shelf exams are a joke in the clinical setting. I've never felt like a bigger idiot spouting **** from academic medicine in clinic.
 
Yeah, evals just predict how good people are at gaming the eval system.

I don't think many people believe that evals measure clinical skills.
 
Shelf exams are a joke in the clinical setting. I've never felt like a bigger idiot spouting **** from academic medicine in clinic.

Don't really understand this point of view. As an example, there are many, many UWorld step 2ck questions (essentially shelf questions) especially in medicine where the question is "What is the next step in the management of this patient? (word for word)". Not sure how you get more clinically oriented than that.
 
  • Like
Reactions: 1 users
Don't really understand this point of view. As an example, there are many, many UWorld step 2ck questions (essentially shelf questions) especially in medicine where the question is "What is the next step in the management of this patient? (word for word)". Not sure how you get more clinically oriented than that.

I'm too cynical to offer any perspective so I'll just shut up.
 
  • Like
Reactions: 1 user
Don't really understand this point of view. As an example, there are many, many UWorld step 2ck questions (essentially shelf questions) especially in medicine where the question is "What is the next step in the management of this patient? (word for word)". Not sure how you get more clinically oriented than that.

As someone who rocked shelf exams and did pretty well (for SDN standards) on step 2, all the while receiving comments such as "performed on a house staff level" on my sub I rotation, I can say it's because you often don't do that specific test or action "next," you do four things at once.

Sent from my SCH-I535 using Tapatalk
 
  • Like
Reactions: 1 users
Once you really understand test writing you'll get that they are trying to get at what is the most important thing or what lowers mortality the most - e.g. You give nitrates, morphine and aspirin at the same time to a guy with chest pain but only aspirin lowers mortality

Source: all honors and 275 ck
 
Once you really understand test writing you'll get that they are trying to get at what is the most important thing or what lowers mortality the most - e.g. You give nitrates, morphine and aspirin at the same time to a guy with chest pain but only aspirin lowers mortality

Source: all honors and 275 ck

No idea who you were directing this at, but the way you brag on this board is hilarious. You really have some kind of fragile ego.

If it was intended for me, I know exactly what they're hinting at, but they say it in a way that is intentionally confusing. What you're stating actually helps to prove my point. They're not judging your clinical knowledge or decision making, they're judging your test taking ability. Just ask, "What initial treatment in acute MI lowers mortality?" instead.
 
  • Like
Reactions: 1 users
or people can stop just being zombies that memorize MONAB and don't think about which one of those drugs actually keeps the patient from croaking versus just provides comfort care

i am pretty awesome
 
or people can stop just being zombies that memorize MONAB and don't think about which one of those drugs actually keeps the patient from croaking versus just provides comfort care

i am pretty awesome

WOW. Congratulations. Beta blockers reduce mortality. As do ACE inhibitors. I guess you forgot?
 
Not as much as aspirin. These things are quantifiable.

I score in the 99th percentile. Don't forget it scrub.
 
Don't hurt your hand from patting yourself so hard on the back there bud.

Sure, there's what's done in the real world and then there's what should be done/what there's evidence for. Just because everyone who walks into the ED who says the word "head" gets a head CT doesn't mean it's the right thing to do. There are also appropriate times to do four things at once though but, to be serious, a lot of what you see in the hospital is just "cover all our bases" kind of stuff. I think you're also forgetting that, in some cases where you would do multiple things at once, the shelves/CK have those options. Am I saying that every shelf question/CK question is super useful? No, but there are quite a few that are.

It's the same argument people have about the Iowa Basics, about the SAT, about the ACT, about the MCAT, about the LSAT, about Step 1/2/3, blah blah blah. "They don't actually test intelligence, they don't test what you need to be a good lawyer, doctor, whatever." The core argument seems to be whether rotation evals are a better "evaluation" of your clinical judgement than shelf exams. I personally (and I believe many students) would answer no. Disregarding all the ass kissing and random chance that your attending/resident likes you or not or is in a bad mood when they fill out your eval, even the parts which are actually related to clinical judgement can be stupidly subjective. Happen to read about pancreatitis the night before while studying for the shelf so you're able to pull Ranson's criteria out of your ass the next day when your team is on call and admit a patient with pancreatitis? Good "clinical judgement". Attending tells you to put more stuff in your differential but doesn't like the fourth diagnosis on your differential when you present the last morning? Bad "clinical judgement".

This is a very good post.

You're right, my comment sounded way more arrogant than I intended... I was trying to basically say that I did well on tests, and I'm good clinically, but I think the tests are intentionally written to confuse people. I understand their angle, and so do most medical students, but what ever happened to being straightforward with questions? If you want to know the most important next action in a trauma patient, it's appropriate to ask for a next step, but in the workup of abdominal pain in a 48 year old male, maybe not so much...

Sent from my SCH-I535 using Tapatalk
 
  • Like
Reactions: 1 user
This is a very good post.

You're right, my comment sounded way more arrogant than I intended... I was trying to basically say that I did well on tests, and I'm good clinically, but I think the tests are intentionally written to confuse people. I understand their angle, and so do most medical students, but what ever happened to being straightforward with questions? If you want to know the most important next action in a trauma patient, it's appropriate to ask for a next step, but in the workup of abdominal pain in a 48 year old male, maybe not so much...

I have a translator that sits by me during every shelf exam.

Not as much as aspirin. These things are quantifiable.

You were saying Aspirin is the only one that helps, which isn't true. Beta blockers and ACE also help. If you're saying which helps the most...well that's another story.
 
this is why you fail at shelf exams and third year, bro. trying to be a lawyer isn't gonna get you a 99. thinking like a med school boss will.

(notice i never said aspirin is the ONLY one that helps, TY! i only said that people spout MONAB without actually knowing which of those helps mortality and which don't... you'll get a lot of n00bz thinking nitrates are super duper important lulz)
 
this is why you fail at shelf exams and third year, bro. trying to be a lawyer isn't gonna get you a 99. thinking like a med school boss will.

(notice i never said aspirin is the ONLY one that helps, TY! i only said that people spout MONAB without actually knowing which of those helps mortality and which don't... you'll get a lot of n00bz thinking nitrates are super duper important lulz)

You dorky, socially awkward kids with no friends who come to SDN acting like a misc troll brah with a "275 CK score" are so easy to spot.

Pipe down there, chief.
 
  • Like
Reactions: 1 user
lol u so jelly

don't be mad cuz you can't get numbers as good as mine
 
lol u so jelly

don't be mad cuz you can't get numbers as good as mine

You're making assumptions about people that are humble. This is often a bad idea.

Just remember, there's always someone who scored better than you. While that's not me, it might be the poster you quoted.

If I ever need a pick me up after your incessant trolling, I'll just go back to the radiology subsection and look at your posts whining about how you didn't get interviews at programs X, Y, Z and how they're all jerks because of it.

Sent from my SCH-I535 using Tapatalk
 
this is why you fail at shelf exams and third year, bro. trying to be a lawyer isn't gonna get you a 99. thinking like a med school boss will.

(notice i never said aspirin is the ONLY one that helps, TY! i only said that people spout MONAB without actually knowing which of those helps mortality and which don't... you'll get a lot of n00bz thinking nitrates are super duper important lulz)

Yeah, my clinical knowledge sucks because I know that there are three drugs in there that lower mortality but don't know which does it best. I will fail when the time comes that a patient presents with acute chest pain but demands I can only use one drug.

Also; its never been said nitro doesn't reduce mortality but that current evidence isnt that strong. That doesn't mean that it doesn't. And how the hell can you even make a statement that nitro isn't important? Gg. No wonder you wanna do radiology; you don't have common sense
 
Awww ur cute, trying to be a lawyer again. too bad you suck at tests since that's what really matters in the game. Program directors care way more about clinical honors than how many ****ty intubations or nurse scut work uve done.
 
Awww ur cute, trying to be a lawyer again. too bad you suck at tests since that's what really matters in the game. Program directors care way more about clinical honors than how many ****** intubations or nurse scut work uve done.


What do the program directors that rejected you care about, then?
 
Scotty has been sent to purgatory.
 
I'm glad I care about learning than silly little numbers or silly grades :p
 
Top