- Joined
- May 9, 2010
- Messages
- 1,186
- Reaction score
- 851
Came across this while looking for something else on the SAEM 2018 website - I can't link directly or view the poster, but go to SAEM18 and expand the Education section, look for "Comparing Standardized Video Interview Scores to Faculty Gestalt"
"Background: The Standardized Video Interview (SVI) is a new assessment tool intended to provide objective data on a residency applicant’s interpersonal communication skills and professionalism. Initial data suggests scores are not correlated with academic competency and provide a new objective measure of applicant attributes. As part of a pilot program, emergency medicine residency programs have agreed to require the SVI in the 2017-2018 residency recruitment cycle. Questions remain, however, about the value of the new data point and how it will correlate with program-specific interview scores.
Objective: To compare SVI scores to faculty gestalt of communication and professionalism.
Methods: As part of the interview process, faculty interviewers were instructed to generate a numerical score for each applicant they met based on their assessment of the applicant’s overall interpersonal communication and professionalism. All interviewers were blinded to the applicant’s SVI score and video. Applicants were scored by faculty on a simplified scale from 1-25; those scores were then converted to the 6-30 scale utilized by the SVI. SVI scores were not used as part of the applicant selection process and faculty were provided no feedback on their assigned scores. All applicants who were attended an interview and submitted an application with an SVI score were eligible for inclusion.
Results: Data was collected for 65 residency interview candidates. Each applicant received a gestalt score from at most 4 faculty interviewers. The average SVI score was 19.88 and the average faculty-generated score was 22.69 (p<0.01). A majority of applicants (80%, n= 52) received higher average scores than their SVI score. SVI scores did not correlate (r=0.08) with faculty gestalt scores.
Conclusions: SVI scores do not appear to measure communication and professionalism in the same way that our faculty do. This could relate to bias introduced by knowledge of an applicant’s academic performance, different types of questions being asked by faculty interviewers, or lack of uniform criteria by which faculty assess these competencies. The next step will be to analyze inter-rater reliability of faculty scores and comparison of these scores to match statistics."
Big thanks to the faculty and residents listed below for taking a thoughtful approach to this apparently useless metric! Truly the heroes we need.
Benjamin Schnapp
– Assistant Residency Program Director, University of Wisconsin, Madison, WI
Daniel Ritter
– Resident, University of Wisconsin Department of Emergency Medicine, Madison, Wisconsin
Aaron Kraut
– Resident, University of Wisconsin Department of Emergency Medicine, Madison, Wisconsin
Sarah Fallon
– Resident, University of Wisconsin Department of Emergency Medicine, Madison, Wisconsin
Keeley Brown
– Resident, University of Wisconsin Department of Emergency Medicine, Madison, Wisconsin
Mary Westergaard
– Resident, University of Wisconsin Department of Emergency Medicine, Madison, Wisconsin
Presenting Author(s)
Emelia Hakes
– Resident, University of Wisconsin Department of Emergency Medicine, Madison, Wisconsin
"Background: The Standardized Video Interview (SVI) is a new assessment tool intended to provide objective data on a residency applicant’s interpersonal communication skills and professionalism. Initial data suggests scores are not correlated with academic competency and provide a new objective measure of applicant attributes. As part of a pilot program, emergency medicine residency programs have agreed to require the SVI in the 2017-2018 residency recruitment cycle. Questions remain, however, about the value of the new data point and how it will correlate with program-specific interview scores.
Objective: To compare SVI scores to faculty gestalt of communication and professionalism.
Methods: As part of the interview process, faculty interviewers were instructed to generate a numerical score for each applicant they met based on their assessment of the applicant’s overall interpersonal communication and professionalism. All interviewers were blinded to the applicant’s SVI score and video. Applicants were scored by faculty on a simplified scale from 1-25; those scores were then converted to the 6-30 scale utilized by the SVI. SVI scores were not used as part of the applicant selection process and faculty were provided no feedback on their assigned scores. All applicants who were attended an interview and submitted an application with an SVI score were eligible for inclusion.
Results: Data was collected for 65 residency interview candidates. Each applicant received a gestalt score from at most 4 faculty interviewers. The average SVI score was 19.88 and the average faculty-generated score was 22.69 (p<0.01). A majority of applicants (80%, n= 52) received higher average scores than their SVI score. SVI scores did not correlate (r=0.08) with faculty gestalt scores.
Conclusions: SVI scores do not appear to measure communication and professionalism in the same way that our faculty do. This could relate to bias introduced by knowledge of an applicant’s academic performance, different types of questions being asked by faculty interviewers, or lack of uniform criteria by which faculty assess these competencies. The next step will be to analyze inter-rater reliability of faculty scores and comparison of these scores to match statistics."
Big thanks to the faculty and residents listed below for taking a thoughtful approach to this apparently useless metric! Truly the heroes we need.
Benjamin Schnapp
– Assistant Residency Program Director, University of Wisconsin, Madison, WI
Daniel Ritter
– Resident, University of Wisconsin Department of Emergency Medicine, Madison, Wisconsin
Aaron Kraut
– Resident, University of Wisconsin Department of Emergency Medicine, Madison, Wisconsin
Sarah Fallon
– Resident, University of Wisconsin Department of Emergency Medicine, Madison, Wisconsin
Keeley Brown
– Resident, University of Wisconsin Department of Emergency Medicine, Madison, Wisconsin
Mary Westergaard
– Resident, University of Wisconsin Department of Emergency Medicine, Madison, Wisconsin
Presenting Author(s)
Emelia Hakes
– Resident, University of Wisconsin Department of Emergency Medicine, Madison, Wisconsin