Abstract: Standardized Video Interview (SVI) score does not correlate with faculty gestalt

This forum made possible through the generous support of SDN members, donors, and sponsors. Thank you.

surely

MD Class of 2018
10+ Year Member
Joined
May 9, 2010
Messages
1,186
Reaction score
851
Came across this while looking for something else on the SAEM 2018 website - I can't link directly or view the poster, but go to SAEM18 and expand the Education section, look for "Comparing Standardized Video Interview Scores to Faculty Gestalt"


"Background: The Standardized Video Interview (SVI) is a new assessment tool intended to provide objective data on a residency applicant’s interpersonal communication skills and professionalism. Initial data suggests scores are not correlated with academic competency and provide a new objective measure of applicant attributes. As part of a pilot program, emergency medicine residency programs have agreed to require the SVI in the 2017-2018 residency recruitment cycle. Questions remain, however, about the value of the new data point and how it will correlate with program-specific interview scores.

Objective: To compare SVI scores to faculty gestalt of communication and professionalism.

Methods: As part of the interview process, faculty interviewers were instructed to generate a numerical score for each applicant they met based on their assessment of the applicant’s overall interpersonal communication and professionalism. All interviewers were blinded to the applicant’s SVI score and video. Applicants were scored by faculty on a simplified scale from 1-25; those scores were then converted to the 6-30 scale utilized by the SVI. SVI scores were not used as part of the applicant selection process and faculty were provided no feedback on their assigned scores. All applicants who were attended an interview and submitted an application with an SVI score were eligible for inclusion.

Results: Data was collected for 65 residency interview candidates. Each applicant received a gestalt score from at most 4 faculty interviewers. The average SVI score was 19.88 and the average faculty-generated score was 22.69 (p<0.01). A majority of applicants (80%, n= 52) received higher average scores than their SVI score. SVI scores did not correlate (r=0.08) with faculty gestalt scores.

Conclusions: SVI scores do not appear to measure communication and professionalism in the same way that our faculty do. This could relate to bias introduced by knowledge of an applicant’s academic performance, different types of questions being asked by faculty interviewers, or lack of uniform criteria by which faculty assess these competencies. The next step will be to analyze inter-rater reliability of faculty scores and comparison of these scores to match statistics."


Big thanks to the faculty and residents listed below for taking a thoughtful approach to this apparently useless metric! Truly the heroes we need.

Benjamin Schnapp
– Assistant Residency Program Director, University of Wisconsin, Madison, WI
Daniel Ritter
– Resident, University of Wisconsin Department of Emergency Medicine, Madison, Wisconsin
Aaron Kraut
– Resident, University of Wisconsin Department of Emergency Medicine, Madison, Wisconsin
Sarah Fallon
– Resident, University of Wisconsin Department of Emergency Medicine, Madison, Wisconsin
Keeley Brown
– Resident, University of Wisconsin Department of Emergency Medicine, Madison, Wisconsin
Mary Westergaard
– Resident, University of Wisconsin Department of Emergency Medicine, Madison, Wisconsin
Presenting Author(s)
Emelia Hakes
– Resident, University of Wisconsin Department of Emergency Medicine, Madison, Wisconsin

Members don't see this ad.
 
  • Like
Reactions: 1 users
Meh.

As skeptical as I am about the SVI, I also recognize that faculty strongly dislike giving low marks (myself included).

As far as I'm concerned, this is evidence for the SVI.
 
Hey I recognize one of those names.
 
  • Like
Reactions: 2 users
Members don't see this ad :)
The SVI is complete garbage and shouldn’t even be a part of the ERAS application. I’ll leave it at that.
 
  • Like
Reactions: 1 users
I’m one of the authors, and unlike the AAMC I’d love for you to AMA.
 
  • Like
Reactions: 1 users
Meh.

As skeptical as I am about the SVI, I also recognize that faculty strongly dislike giving low marks (myself included).

As far as I'm concerned, this is evidence for the SVI.

It'd be one thing if the report came out and said that faculty scores correlated with the SVI score but tended to be higher on average... but they were both higher and seemingly unrelated to the SVI score.

The faculty didn't watch the video. The faculty met the students and interviewed them, then assigned them a score based on the in-person interview. Isn't that how most residency interviews are conducted, with the ultimate goal of ranking candidates in mind? Are you saying that the interview portion of the ranking process is bogus because faculty strongly dislike giving low marks?
 
It'd be one thing if the report came out and said that faculty scores correlated with the SVI score but tended to be higher on average... but they were both higher and seemingly unrelated to the SVI score.

The faculty didn't watch the video. The faculty met the students and interviewed them, then assigned them a score based on the in-person interview. Isn't that how most residency interviews are conducted, with the ultimate goal of ranking candidates in mind? Are you saying that the interview portion of the ranking process is bogus because faculty strongly dislike giving low marks?

I actually don't think that I can say based on the abstract alone. If the rank order of the applicants was significantly different between the modalities, than that result is more surprising than the faculty scores simply being higher. Were the rank orders also quite different?
 
I actually don't think that I can say based on the abstract alone. If the rank order of the applicants was significantly different between the modalities, than that result is more surprising than the faculty scores simply being higher. Were the rank orders also quite different?
Overall rank order isn't determined solely on how the interview goes, though. It takes into consideration things like scores, research, extracurriculars, and above all, SLOEs. So I don't think it's fair to compare rank order to faculty score or SVI score.
 
Overall rank order isn't determined solely on how the interview goes, though. It takes into consideration things like scores, research, extracurriculars, and above all, SLOEs. So I don't think it's fair to compare rank order to faculty score or SVI score.

Agreed - I meant, if you ranked the SVI scores and the faculty gestalt scores, did the interviewees end up in roughly the same order from "worst" to "best"?

If so, I think my initial interpretation is supported. If not, then I will retract it.
 
Agreed - I meant, if you ranked the SVI scores and the faculty gestalt scores, did the interviewees end up in roughly the same order from "worst" to "best"?

If so, I think my initial interpretation is supported. If not, then I will retract it.
In the abstract, it says the two scores didn't correlate (r=0.08). So nope, the interviewees did not end up in roughly the same order from worst to best.
 
  • Like
Reactions: 1 user
In the abstract, it says the two scores didn't correlate (r=0.08). So nope, the interviewees did not end up in roughly the same order from worst to best.
Exactly there was no correlation between the gestalt scores (just professionalism and communication) and the overall interview scores and the SVI scores.

I think the thing to bear in mind is that @WilcoWorld isn't necessarily wrong. What our data tells us is that the assessment provided by the SVI is different from the assessment we get from a traditional interview. It could turn out that the SVI is a better predictor of resident performance or it could not.
 
  • Like
Reactions: 1 users
In the abstract, it says the two scores didn't correlate (r=0.08). So nope, the interviewees did not end up in roughly the same order from worst to best.

s'pose I could've figured that out if I'd read the abstract title...thanks for the discussion
 
  • Like
Reactions: 1 user
I attended SAEM this year and there was a rather long presentation at the medical student symposium from the representative at AAMC who is in charge of the SVI. They did some analysis and surveys of SVI scores and here are the key things that stood out to me:
1. Program directors sparingly used SVI in ranking candidates this year. Probably because it was a pilot thing.
2. A low SVI score was taken into consideration more than a high SVI score in calling candidates for interview.
3. Program directors had access to the original video recordings of candidates. Most PDs wanted to watch the original video recordings of the high SVI scoring candidates.
4. And this is the key thing. Almost everyone agreed how closely the candidate's original SVI recording matched with how the candidate was in the in-person interview. Which means that the SVI may have a utility at least in the coming years and it is here to stay. Even though most PDs answered they will use SVIs sparingly in the upcoming match cycle.

Considering the fact that the SVI is scored by clinical psychologists with a great deal of inter-scorer agreement, I can see this becoming a thing.

Edit : I am recalling this from memory and cannot guarantee the accuracy of the statements I made.
 
Considering the fact that the SVI is scored by clinical psychologists with a great deal of inter-scorer agreement, I can see this becoming a thing.

Huh, I thought it was scored by "HR professionals" - at least, that's what HireVue, the company conducting the SVI, said. I know you said you're reciting this from memory, but did you happen to read the psychologists part somewhere?

For what it's worth, I talked to someone in program leadership who stated his faculty watched many/most/all of the videos and were unable to differentiate between high scoring videos and low scoring interviews, which is theoretically consistent with what the abstract above says. So I'm surprised faculty at SAEM said they saw a correlation in score and applicant quality.

Edit: Wait, were the faculty at SAEM saying that the video recording itself is what was similar to the in-person interview, even though the score wasn't helpful? If so... I mean, that makes the SVI much less useful, if leadership have to actually watch the videos in order to glean information about the candidate. And it's dangerous to have an unuseful score attached to a useful video, since people will be tempted to put stock in the score instead of investing the time to watch hundreds of videos.
 
The SVI is worthless and a waste of time for applicants and program leadership. Let’s be real...pretty much no one used this to decide who to invite to interview or rank on their match list. It’s sad that the SVI caused our specialty to spend valuable research dollars on investigating how worthless this part of the application is when they could have been spent on something related to patient care.
 
  • Like
Reactions: 3 users
Considering the fact that the SVI is scored by clinical psychologists
No it’s not. It’s HR reps who took an hour course on what it's like to be an ED resident.
DbkhBGPUQAAvv0Q.jpg
 
  • Like
Reactions: 1 user
Edit: Wait, were the faculty at SAEM saying that the video recording itself is what was similar to the in-person interview, even though the score wasn't helpful? If so... I mean, that makes the SVI much less useful, if leadership have to actually watch the videos in order to glean information about the candidate. And it's dangerous to have an unuseful score attached to a useful video, since people will be tempted to put stock in the score instead of investing the time to watch hundreds of videos.
At the AAMC update at CORD (can't speak to SAEM) several people mentioned a stark difference between the video and in-person interview. The answer to that is quite simple: we've already started coaching the SVI like we coach Step 2 CS. It will be come formulaic and useless. I've even heard of some programs teaming with the business and theater schools at their institution to prep students. I mean, just look at this graphic from the AAMC:
Dbkb1ISWkAAw1kq.jpg

50% of students made up their answers!
 
  • Like
Reactions: 1 user
50% of students made up their answers!

Not surprising, since most applicants have little life experience to work from. As a non-traditional student who worked as a physician assistant for a number of years, even I had difficulty with questions (along the lines of - since we can't give out specific questions) "describe an instance of how you responded when you saw religious or racial bias between a supervising physician and a patient".

I also heard that someone who scored highly on the SVI made a number of short responses like, "I never saw this or experienced this." And after a short sentence like that clicked to end the interview response. This may just be rumor.
 
I also heard that someone who scored highly on the SVI made a number of short responses like, "I never saw this or experienced this." And after a short sentence like that clicked to end the interview response. This may just be rumor.

If this is true let me know, asking for a friend :D
 
CORD put out a new blog post with some fun facts about how the SVI was used last year: The Standardized Video Interview: Updates for the 2018-2019 Application Season


Key points, emphasis mine:

"At the Academic Assembly, Ms. Overton further offered some data on how Program Directors used SVI data for the initial year of the pilot. She presented that 70% of Program Directors indicated that “SVI scores were not important in deciding whom to invite to the in-person interview.” Approximately 80% of Program Directors watched at least one SVI video, with a very wide range of usage both in terms of how many videos were watched and how much of each video was viewed. Ms. Overton noted that programs were more likely to view videos from applicants with a higher Step 1 score or applicants with a higher SVI score. She presented AAMC data that there were no differences in views for applicants by race, ethnicity, or gender. She also included that only 57% of Program Directors indicated that they would be “somewhat or more likely to use [SVI score] in ERAS 2019 selection.”

In an abstract presented at this year’s Society of Academic Emergency Medicine (SAEM) conference in Indianapolis, Dr. Melissa White presented data obtained from nine ACGME residency programs showing that when SVI scores were considered in addition to traditional review of ERAS applications for approximately 2100 students, they did not change the likelihood of being interviewed in 93% of cases. While the score was equally likely to increase or decrease the likelihood of being invited, on subgroup analysis, SVI scores in the 6-11 range decreased the likelihood of being invited by 64%."
 
Top