Is there evidence for using Residency Interviews?

[openheartpsychiatrist]

---

Members don't see this ad.

I understand the usefulness of a residency interview in order to see a city, a hospital, maybe even to establish 'fit' but I was wondering if there is evidence out there that correlates morbidity and mortality or performance with residency interviews? It has been a big topic of discussion among my friends and colleagues and if anyone has evidence for or against the use of interviews, I'd love to hear/see it. This is not me saying that I am for or against interviews but I would love an academic discussion of their value.

Gratefully

OHP

 

[MacDonaldTriad]

Academic Psychiatry

March 2005, Volume 29, Issue 1, pp 69-75

First online: 09 January 2014

The Relationship Between Psychiatry Residency Applicant Evaluations and Subsequent Residency Performance

• Karon DawkinsAffiliated withDepartment of Psychiatry, University of North Carolina-Chapel HillEmail author
• , R. David EkstromAffiliated withDepartment of Psychiatry, University of North Carolina-Chapel Hill
• , Allan MaltbieAffiliated withDepartment of Psychiatry, University of North Carolina-Chapel Hill
• , Robert N. GoldenAffiliated withDepartment of Psychiatry, University of North Carolina-Chapel Hill

 

[Doctor Bagel]

Academic Psychiatry

March 2005, Volume 29, Issue 1, pp 69-75

First online: 09 January 2014

The Relationship Between Psychiatry Residency Applicant Evaluations and Subsequent Residency Performance

• Karon DawkinsAffiliated withDepartment of Psychiatry, University of North Carolina-Chapel HillEmail author
• , R. David EkstromAffiliated withDepartment of Psychiatry, University of North Carolina-Chapel Hill
• , Allan MaltbieAffiliated withDepartment of Psychiatry, University of North Carolina-Chapel Hill
• , Robert N. GoldenAffiliated withDepartment of Psychiatry, University of North Carolina-Chapel Hill

So I guess the answer so far is no (at least based on just looking at the abstract). Interviews in general are not great tools for figuring out who would do a good job at something. Interviews I guess let you weed out the train wrecks, though (or at least some of them). My program had one total train wreck resident -- from what I heard from the attendings, no one actually remembered this person from interviews, and they consequently wound up somewhere on the list (not at the top but not at the absolute bottom either). I remember interviewing people for a position on a journal that I worked on a long time ago -- the people I thought would be awesome wound up kind of sucking, and the people I was worried about (we weren't too selective) wound up doing well.

 

[MacDonaldTriad]

For every resident who thinks their program deceived them and didn't admit to everything, there are two programs that feel the inverse is true. Anyway, it is clear we have little or no ability to predict graduate performance. This may suggest that we lessen the importance of past performance, but what then do we base things on?

 

[MidwestMD2015]

I'm also interested to know interview/performance correlations, but I'm not aware of data on this and a cursory search wasn't very revealing!!
First, seems like it would be valuable to establish the reliability of an interview or system of interviews.
Medical schools, having more consistent experience with larger numbers of people, have been working on this for a while.

The 'latest and greatest' for the recent years has been the MMI (Multiple Mini-Interview) format for medical school admissions.
The foundation of the system is based on experience first at McMaster University, and there has been much subsequent research and discussion.
Here is a link to the paper, "An admissions OSCE: the multiple mini-interview"

That's at least something to consider. There are discussions about this issue at the AAMC from a UME perspective, but for GME it's much more challenging due to the vast differences between different specialties and different programs.

The value of a single interview, as a data point to predict something about a person, is really interesting to think about and I'm pretty skeptical. Wonder what it would be like without them? Are other factors more reliable in predicting outcomes? Since interviews will continue to happen for the foreseeable future, are there ways to make them more reliable as predictors (ie., would a MMI for residency be useful?) Are there things that happen in psychiatry interviews that don't in other specialties? More questions than answers!

 

[Doctor Bagel]

So, I guess the question that follows is so why does the our community, which values evidence, use interviews? They cost so much money and are so troublesome. Is there any evidence to the contrary that interviews actual increase mortality and morbidity or performance?

Still would love more literature to read!

I was on my medical school admissions committee and I was an interviewer. I would like to think that we weeded out a couple of really bad apples but I have no PROOF of that.

OHP

These days, you could also easily replace in person interviews with Skype style interviews, which would be much cheaper for all parties involved. Interviews certainly are costly especially when you consider that they are often funded on student loans which have relatively high interest rates.

 

[notdeadyet]

So, I guess the question that follows is so why does the our community, which values evidence, use interviews? They cost so much money and are so troublesome. Is there any evidence to the contrary that interviews actual increase mortality and morbidity or performance?

Unless I'm mis-reading this article (always a risk), my take is that this article doesn't have anything to do with evaluating the effectiveness of interviews. This article merely looks at how effective they are at gauging future performance among matriculants. Studying how well you think your incoming class is going to perform before they start and tracking them afterwards has little/nothing to do with interviews.

Interviews are helpful for identifying potential superstars, but these folks tend to present well on paper. Where interviews are helpful is to discover hidden train wrecks, who may look great on paper but can often present themselves poorly in interview.

This is the same in any field. The resume is great, but you want the applicant in front of you to get a feel for how they will be to work with. If they come across as someone who is going to be a problem, you go with someone else.

 

[downwithDTB]

Remember that the mentioned study only evaluates performance within the very limited range of applicants who actually matched to UNC, therefore, if the goal of interviews is to filter out the really bad, this study couldn't detect that. It just says that most of the noise that matches to a decent program is noise. We simply don't know if the interview processes weeded out really bad apples.

Even within this context, three important questions exist: 1) Was resident performance evaluated in a meaningful way allowing for comparisons to be made/do we even have a meaningful metric 2) UNC appears to be mostly US MDs. Are interviews more useful for evaluating DOs/IMG/FMGs given that PDs may be less familiar with their programs? 3) Is there a purpose for interviews that goes beyond evaluation of potential competence? For example, evaluation of personality/program "fit"/philosophy/interest?

 

[Mad Jack]

I understand the usefulness of a residency interview in order to see a city, a hospital, maybe even to establish 'fit' but I was wondering if there is evidence out there that correlates morbidity and mortality or performance with residency interviews? It has been a big topic of discussion among my friends and colleagues and if anyone has evidence for or against the use of interviews, I'd love to hear/see it. This is not me saying that I am for or against interviews but I would love an academic discussion of their value.

Gratefully

OHP

Who cares whether they improve outcomes- interviews make sure you're picking residents that you're more likely to be able to stand dealing with for a few years. Most US grads will likely perform similarly outcome-wise, but a good interview can make sure you're picking someone that not only goes on to do well, but that you don't throw them out of the program before they can get the chance because they grate on your nerves or don't fit in.

 

[Doctor Bagel]

Who cares whether they improve outcomes- interviews make sure you're picking residents that you're more likely to be able to stand dealing with for a few years. Most US grads will likely perform similarly outcome-wise, but a good interview can make sure you're picking someone that not only goes on to do well, but that you don't throw them out of the program before they can get the chance because they grate on your nerves or don't fit in.

We tend to like people who are attractive and also who are like us, so interviews are certainly a way to perpetuate discrimination and reinforce insularity in training settings and in medicine in general. I don't believe that I have any ability in interviewing someone to know who would be a good candidate for a job (or even who would actually in the long run be a pleasant person to work with -- likeability and attractiveness don't necessarily correlate with doing your job, supporting others, etc.). There's a reason who people are moving more toward structured interviews -- evidence for unstructured interviews is pretty limited.

 

[Mad Jack]

We tend to like people who are attractive and also who are like us, so interviews are certainly a way to perpetuate discrimination and reinforce insularity in training settings and in medicine in general. I don't believe that I have any ability in interviewing someone to know who would be a good candidate for a job (or even who would actually in the long run be a pleasant person to work with -- likeability and attractiveness don't necessarily correlate with doing your job, supporting others, etc.). There's a reason who people are moving more toward structured interviews -- evidence for unstructured interviews is pretty limited.

It doesn't help you sort in who will be good at the job, but it helps you sort out someone that might be bad for your department. It's like a sensitivity/specificity sort of thing- great for the rule-outs, but not so much for the rule-ins.

 

[scudiera]

I agree with you, personally. Just playing Devil's Avocado, where is the evidence that it will take out those who are bad for a department? I mean how many people a year to program directors "rule out" based on an interview? What would they have to do to be "ruled out"? I am all for structured interviews.

1. That's a delicious typo.
2. Even if you rule out just one, some would argue it's worth it. Someone who makes a lot of excuses during their interview/passes the buck, somebody who's horribly awkward in person or uncomfortable interacting with residents, maybe just somebody who lacks all social awareness or is singularly focused on one thing and you realize their focus is either not a good fit or so narrow that they're bound to be disappointed/may not be interested in the field as a whole. Just ideas.

 

[Doctor Bagel]

1. That's a delicious typo.
2. Even if you rule out just one, some would argue it's worth it. Someone who makes a lot of excuses during their interview/passes the buck, somebody who's horribly awkward in person or uncomfortable interacting with residents, maybe just somebody who lacks all social awareness or is singularly focused on one thing and you realize their focus is either not a good fit or so narrow that they're bound to be disappointed/may not be interested in the field as a whole. Just ideas.

Shouldn't these things also be captured by a student's dean's letter? So far, it seems that the argument is interviews might have a role in weeding out the horrible but aren't so useful in differentiating between the non-horrible (99% of applicants). Of course they don't necessarily weed out the horrible because again, we had a horrible (like really horrible) resident who was apparently totally bland and average on interview day like almost everyone else.

 

[MacDonaldTriad]

Dean's letters are fairly watered down. What is objective is objective and formal discipline is mentioned, but everything else isn't. Medical schools try and put a good lite on their finished products. You will never see a sentence like:
"This guy got though OK, but our selection process really missed the mark on this guy."

 

[whopper]

Per studies in Industrial Psychology interviews and LORs have no correlation with job performance.
I personally do find LORs useful if you know the writer of the letter and know they will only write a letter for someone if they think that person is good. Problem is 99% of the time you don't know who the writer is.

It's a shame. The only thing that does seem to have a correlation are standardized exams and that by itself is a poor use of criterion to accept someone.

 

[downwithDTB]

I am all for structured interviews.

I haven't seen the literature on structured interviews, nor have I actually experienced one; however, I imagine that if they become the norm for med school and/or residency interviews, that formal training sessions/review courses will be the logical answer. Mock interviews now are of little value, but if you have a rubric, known questions and an expert teacher, I doubt structured interviews can't be trained for.

For what it's worth: when interviewing people for med school, I found that challenging people on the spot, usually related to their research, identified a good number of people who were pretty clueless. Taking time to ask a lot of pointed "why" questions and going in with some expertise and preparation (e.g. reading an applicants papers and speaking with someone in their field), might be more beneficial than the typical interview (at least for candidates claiming a special interest or experience), which is hardly challenging. Of course, this is hypothesis.

 

[Rogue Penguin]

Where interviews are helpful is to discover hidden train wrecks, who may look great on paper but can often present themselves poorly in interview.

It seems like even if someone comes across as a train wreck during an interview, you'll never really know if your program dodged a bullet or not. It's not like you get to experience having them as a resident after you reject them. I wonder if commonly rejected applicants go on to cause problems at wherever they match (assuming they match at all).

 

[Shikima]

I guess the question is really about the match process once it's watered down. In actuality, no one really cares about the interview process as we're all interviewing for a job. The question and concern is then the matching process as it is vague and ambiguous despite how it is mathematically derived.

 

[notdeadyet]

I personally do find LORs useful if you know the writer of the letter and know they will only write a letter for someone if they think that person is good. Problem is 99% of the time you don't know who the writer is.

This is why I never buy into the "it doesn't matter where you go to medical school argument."

LORs always have the most weight when written by someone we know. This is why you have so much network or regional matches.

 

[notdeadyet]

It seems like even if someone comes across as a train wreck during an interview, you'll never really know if your program dodged a bullet or not. It's not like you get to experience having them as a resident after you reject them.

Train wrecks de interview are not folks who don't interview well. Some folks are prone to butterflies, nervous jitters, etc.

The train wrecks are the raging narcissists, egomaniacs, and other character traits that you do not want to see in any prospective intern. Basically, if these folks were bugs, they'd be the ones with the fluorescent yellow bellies.

 

[hamstergang]

I guess the question is really about the match process once it's watered down. In actuality, no one really cares about the interview process as we're all interviewing for a job. The question and concern is then the matching process as it is vague and ambiguous despite how it is mathematically derived.

This post makes no sense to me. The match process, being a rather simple to describe algorithm, is about as unambiguous and clear as you can get. And this thread isn't about the match. It clearly is asking about the utility of interviewing for determining how good or bad a resident will be. So this post just seems odd to me and it makes me wonder if you intended something other than what I interpreted.

 

[Shikima]

No. Not odd. Just a different way I was interpreting the process. Every person interviews for a job. Even a janitor. It's not the interview that's in question because it doesn't matter the job or the environment in which the interview is taking place. The attempt is to convenience a large number of people rather than some Fortune 500 companies that have 4-5-6 interviews for 1 or 2 positions.

 

[hamstergang]

It's not the interview that's in question

But it really is. It's the job that's irrelevant. Do in-person interviews actually add value in helping applicants and employers make better decisions? That's what's being asked. The match has nothing to do with it.

 

[Shikima]

But it really is. It's the job that's irrelevant. Do in-person interviews actually add value in helping applicants and employers make better decisions? That's what's being asked. The match has nothing to do with it.

This is what I'm saying. Show me a job you're applying for that never interviews a person.

Go on, I dare you!

I double-dog dare you!

 

[downwithDTB]

This is what I'm saying. Show me a job you're applying for that never interviews a person.

Go on, I dare you!

I double-dog dare you!

US Military enlisted positions?

 

[OldPsychDoc]

US Military enlisted positions?

They're more selective than you think--definitely looking to weed out 'bad elements'.

 

[hamstergang]

This is what I'm saying. Show me a job you're applying for that never interviews a person.

Go on, I dare you!

I double-dog dare you!

So you agree that this has nothing to do with the match. This is a question about the utility of interviews for jobs, period.

 

[Shikima]

US Military enlisted positions?

Recruiters are doing interviews. On the phone, in person multiple times with different people, and the again at MEPS.

 

[Shikima]

So you agree that this has nothing to do with the match. This is a question about the utility of interviews for jobs, period.

No... I said it was the match that was ambiguous despite the results that create a ranking list which are derived from interviews. The one detailed problem I have with interviews is that it's like a first date, everyone is making assumptions by putting on their best act/behavior to find or be a selectee.

 

[Shikima]

So my next question would be it seems like we mostly agree that interviews aren't very predictive of future performance. Is there anyone who thinks that interviews are actually more harmful than say, just picking the person with the highest scores. Once again, I am absolutely not advocating that, I'm just curious about opinions!

Speed dating?