Residents’ perspectives on the introduction, conduct and value of the Objective Structured Clinical Examination-Results of the 1st nationwide questio

[deleted1101762]

Passing the Objective Structured Clinical Examination (OSCE) is currently a requirement for the majority of candidates to gain American Board of Anesthesiology (ABA) initial certification. Many publications from the ABA have attempted to justify its introduction, conduct and value. However, the ABA has never attempted to understand the views of residents.

A total of 4237 residents and fellows at various training levels from 132 programs were surveyed by asking to fill a Google questionnaire prospectively between March 8th, 2021 and April 10th, 2021. Every potential participant was sent an individual email followed by 2 reminders.

The overall response rate was 16.75% (710 responses to 4237 invitations). On a 5-point Likert scale with 1 as “very inaccurate” and 5 as “very accurate,” the mean accuracy of objective structured clinical examination (OSCE) in assessing communication skills and professionalism was 2.33 (1.00) and 2.14 (1.00) respectively. In terms of the usefulness of OSCE training for improving physicians’ clinical practice, avoiding lawsuits, teaching effective communication with patients and teaching effective communication with other providers, the means on a 5-point Likert scale with 1 as “Not at all useful” and 5 as “Very useful” were 1.86 (1.04), 1.69 (0.88), 1.79 (1.02), and 1.82 (1.07) respectively. Residents unanimously thought that factors such as culture, race/ethnicity, religion and language adversely influence the assessment of communication skills. On a 5-point Likert scale with 1 as “not at all affected” and 5 as “very affected,” the corresponding scores were 3.45 (1.20), 3.19 (1.26), 3.89 (1.30), and 3.18 (1.12) respectively. Interestingly, nationality and political affiliation were also thought to influence this assessment, however, to a lesser extent 2.40 (1.26) and 2.28 (1.26). In addition, residents believed it is inappropriate to test non-cardiac anesthesiologists for TEE skills 2.39 (1.10), but felt it was appropriate to test non-regional anesthesiologists in Ultrasound skills 3.29 (1.21). Lastly, nearly 80% of the residents think that money was the primary motivating factor behind ABA’s introduction of the OSCE. Over 96% residents think that OSCE should be stalled, either permanently scrapped (60.8%) or paused (35.8%).

CONCLUSIONS: Anesthesiology residents overwhelmingly indicated that the OSCE does not serve any useful purpose and should be immediately halted.

---

Members don't see this ad.

 

[Ronin786]

Submit to A&A?

 

[deleted1101762]

Rejected, this is the response
A senior member of our Editorial Board and I have pre-reviewed your manuscript. Based on this pre-review, I have to decline to receive your manuscript for publication in our Journal.





The rationale for this decision is as follows:

1. The survey is designed in the way that clearly shows that the authors of the survey and article do not believe in the OSCE.

1. We do not believe that trainees are authorities on what they will need to know when they graduate from their training program. It would have been better to survey actual graduates. However, it would have been difficult for 2 reasons; 1) the OSCE has only recently been introduced; and 2) difficulty in obtaining a list of names/addresses to carry out the survey.

1. Mail/email surveys with low response rates are difficult to interpret and have the bias that those that are doing poorly are more likely to be critical of how they are graded or judged.

I would suggest to possibly send your article to the journal Academic Medicine.





Thank you for considering our Journal for your work.

 

[MirrorTodd]

Lol that WAS a fairly predictable response.

 

[deleted1101762]

Lol that WAS a fairly predictable response.

Absolutely, I am certain someone will take it.

 

[coffeebythelake]

Rejected, this is the response
A senior member of our Editorial Board and I have pre-reviewed your manuscript. Based on this pre-review, I have to decline to receive your manuscript for publication in our Journal.





The rationale for this decision is as follows:

1. The survey is designed in the way that clearly shows that the authors of the survey and article do not believe in the OSCE.

1. We do not believe that trainees are authorities on what they will need to know when they graduate from their training program. It would have been better to survey actual graduates. However, it would have been difficult for 2 reasons; 1) the OSCE has only recently been introduced; and 2) difficulty in obtaining a list of names/addresses to carry out the survey.

1. Mail/email surveys with low response rates are difficult to interpret and have the bias that those that are doing poorly are more likely to be critical of how they are graded or judged.

I would suggest to possibly send your article to the journal Academic Medicine.





Thank you for considering our Journal for your work.

And what support do they have to implement thr osce in the first place? Seems like the burden should be on the ABA to qualify its effectiveness in making better clinicians (the evidence doesn't exist)

 

[coffeebythelake]

Seems like the ABA like to grasp at low hanging fruit to fill their cofers going after new grads seeking certification. All in thr name of quality and patient safety... what a joke. Maybe they should recall and reassess the skill level for all those grey haired armchair anesthesiologists signing charts who haven't intubated a patient in 10 years?

 

[Dave1980]

your response rate is really low and will skew towards people unhappy with the system. I have to agree with the reviewer in terms of study design. kind of a garbage article

 

[deleted1101762]

your response rate is really low and will skew towards people unhappy with the system. I have to agree with the reviewer in terms of study design. kind of a garbage article

What do you think is a good response rate for an online survey? our study has an error of less than 4% for a 95% confidence. Typically online surveys get a rerponse of about 5-10%.

 

[deleted1101762]

Seems like the ABA like to grasp at low hanging fruit to fill their cofers going after new grads seeking certification. All in thr name of quality and patient safety... what a joke. Maybe they should recall and reassess the skill level for all those grey haired armchair anesthesiologists signing charts who haven't intubated a patient in 10 years?

I guess the following graph does not need an explanation

 

[Dave1980]

What do you think is a good response rate for an online survey? our study has an error of less than 4% for a 95% confidence. Typically online surveys get a rerponse of about 5-10%.

Your calculated error rate and confidence is only for the 15% of people who responded. This 15% is unlikely to reflect the overall sentiment of the group.

Your response rate might be OK for an online survey but there are many good reasons why online surveys don't get published in academic journals. The study design is fatally flawed from the beginning.

 

[deleted1101762]

your response rate is really low and will skew towards people unhappy with the system. I have to agree with the reviewer in terms of study design. kind of a garbage article

Do you think these graphs will change much if all the 4000+ residents reply

 

[Dave1980]

Do you think these graphs will change much if all the 4000+ residents reply View attachment 335043

Yes I do. Sorry that you spent so much time on this project and they rejected it. I don't have a dog in this fight but the study design is terrible and this kind of thing should not be published in an academic journal.

 

[deleted1101762]

Your calculated error rate and confidence is only for the 15% of people who responded. This 15% is unlikely to reflect the overall sentiment of the group.

Your response rate might be OK for an online survey but there are many good reasons why online surveys don't get published in academic journals. The study design is fatally flawed from the beginning.

What about this? Sun H, Chen D, Warner DO, Zhou Y, Nemergut EC, Macario A, Keegan MT. Anesthesiology Residents' Experiences and Perspectives of Residency Training. Anesth Analg. 2021 Apr 1;132(4):1120-1128. doi: 10.1213/ANE.0000000000005316. PMID: 33438965., an internal survey with 36% response, would you say the same?

 

[deleted1101762]

Yes I do. Sorry that you spent so much time on this project and they rejected it. I don't have a dog in this fight but the study design is terrible and this kind of thing should not be published in an academic journal.

This is how I calculated the error before sending questionnaire, by the way, it will get to a journal with better impact factor Margin of Error Calculator | SurveyMonkey

 

[deleted1101762]

Yes I do. Sorry that you spent so much time on this project and they rejected it. I don't have a dog in this fight but the study design is terrible and this kind of thing should not be published in an academic journal.

You should wait to read the entire paper before calling the responses of 710 US residents garbage.

 

[Dave1980]

What about this? Sun H, Chen D, Warner DO, Zhou Y, Nemergut EC, Macario A, Keegan MT. Anesthesiology Residents' Experiences and Perspectives of Residency Training. Anesth Analg. 2021 Apr 1;132(4):1120-1128. doi: 10.1213/ANE.0000000000005316. PMID: 33438965., an internal survey with 36% response, would you say the same?

There are a lot of bad articles that make their way into the literature. They also had over 2x the response rate you did.

 

[deleted1101762]

There are a lot of bad articles that make their way into the literature. They also had over 2x the response rate you did.

There's is internal survey and as a result, it is pretty low, mine is external. However, I Do respect your opinion.

There are a lot of bad articles that make their way into the literature. They also had over 2x the response rate you did.

This paper of mine will answer how papers get into high impact factor journals! It could not get into a "good" journal for a reason

Possible Bias in the Publication Trends of High Impact Factor Anesthesiology and Gastroenterology Journals –An Analysis of 5 Years' Data
​

Background:​

We hypothesize that being an editorial board member (EBM) in a high impact factor specialty medical journal increases the chances of publishing in the same journal.

Materials and Methods:​

The publication trends of the first five EBMs in the five highest impact factor Anesthesiology and Gastroenterology journals were analyzed. Preceding 5 years' publications appearing on PubMed were grouped into as follows: number of publications in the journal in which the EBM serves (N1), number of publications by the same author in the other four highest impact factor (IF) journals (N2) and number of publications in all the other journals (N3). We evaluated the probability of the observed distribution of publications in the five highest IF journals happening by chance alone, assuming that all the EBMs had the same opportunity of publishing in any of these journals. The probability of publishing in their own journal was assumed to be one fifth.

Results:​

The EBMs published their manuscripts in their own journal at a very high frequency. Encompassing all ten journals, the calculated P value for such a distribution was <0.001. In two journals, Anesthesia and Analgesia and Anaesthesia, the EBMs' publications in their journal were more than twice the cumulative total in the remaining four journals. In three of the five gastroenterology journals analyzed, combined publications of the five EBMs were greater in their own journal than the remaining four journals combined.

Conclusions:​

Despite proclaimed fair peer review process, EBMs seem to get preference in their own journals.
Keywords: Anesthesiology, bias, editorial board, ethics, evidence, gastroenterology

 

[Mman]

while I personally think OSCE type stuff is just stupid in terms of graduating residency, I also do not care one iota what current residents opinion of it is. Their opinion is meaningless.

But just because I agree with this small survey does not mean it is relevant.

 

[deleted162650]

Ironically, I think stupid BS exams like the OSCE will actually become relevant/necessary with the proliferation of low quality community based HCA type residency programs.

 

[anbuitachi]

sucks it didnt get accepted,
but like above, i do agree there is bias in surveying residents since they have an incentive to have the test cancelled.

 

[coffeebythelake]

sucks it didnt get accepted,
but like above, i do agree there is bias in surveying residents since they have an incentive to have the test cancelled.

What clinical deficit in training exists in tge first place to create a new osce testing

 

[GassYous]

sucks it didnt get accepted,
but like above, i do agree there is bias in surveying residents since they have an incentive to have the test cancelled.

They just came up with it as a money grab at the same time as step 2 cs is getting canceled.

 

[vector2]

What clinical deficit in training exists in tge first place to create a new osce testing

I would bet that compared to 10 years ago there are many more programs that have shtty or nonexistent didactics, a dearth of fellowship trained staff, residents barely meeting case minimums and when they do it's community hearts, low risk OB, healthy kids, always using the glidescope instead of a fiberoptic, never attempting a thoracic epidural, wedge resections and not pneumonectomies etc.......i.e. not doing tertiary cases and advanced techniques.

Now I don't think the OSCE is necessarily the solution to this clinical deficit, but there is a deficit there since the ACGME clearly does not GAF about minimum standards.

 

[coffeebythelake]

I would bet that compared to 10 years ago there are many more programs that have shtty or nonexistent didactics, a dearth of fellowship trained staff, residents barely meeting case minimums and when they do it's community hearts, low risk OB, healthy kids, always using the glidescope instead of a fiberoptic, never attempting a thoracic epidural, wedge resections and not pneumonectomies etc.......i.e. not doing tertiary cases and advanced techniques.

Now I don't think the OSCE is necessarily the solution to this clinical deficit, but there is a deficit there since the ACGME clearly does not GAF about minimum standards.

Did the acgme dumb down their requirements for case minimums? As you've said this doesn't seem like an osce centered solution.

 

[deleted1101762]

Did the acgme dumb down their requirements for case minimums? As you've said this doesn't seem like an osce centered solution.

Professionalism
along with interpersonal and communication skills are two elements of the six Competency-Based
Medical Education attributes advocated by the Accreditation Council for Graduate Medical
Education (ACGME) and tested in the OSCE. In part, the OSCE was introduced to
satisfy the 2015 ACGME safety and quality improvement requirements for all residency programs,
although, the ACGME did not specify that the OSCE should be used to test these requirements. In fact,
the ACGME did not mandate any testing at the certification point.

 

[deleted875186]

Let’s face it, the ABA created the OSCE without any data, justification, or reasoning other than to collect more money. No data or evidence that it is useless is going to get rid of the test.

 

[anbuitachi]

They just came up with it as a money grab at the same time as step 2 cs is getting canceled.

i agree its a money grab and its useless

 

[coffeebythelake]

Professionalism
along with interpersonal and communication skills are two elements of the six Competency-Based
Medical Education attributes advocated by the Accreditation Council for Graduate Medical
Education (ACGME) and tested in the OSCE. In part, the OSCE was introduced to
satisfy the 2015 ACGME safety and quality improvement requirements for all residency programs,
although, the ACGME did not specify that the OSCE should be used to test these requirements. In fact,
the ACGME did not mandate any testing at the certification point.

Did professionalism plummet all of a sudden, to warrant an osce for new grads?

 

[Mman]

Did professionalism plummet all of a sudden, to warrant an osce for new grads?

perhaps

I spoke within the last 12 months to a program director of a top 10 program (that I did not attend) who lamented the work ethic of residents these days. They implied it was an act of god to try to get them to stay in the OR past 3 PM and none of their graduates were looking for jobs that required significant call, even if the pay was good. They would rather take mommy track pay and mommy track hours than work for the big bucks.


I have no idea of how to judge the claims, but they had no reason to lie to me.

 

[abolt18]

perhaps

I spoke within the last 12 months to a program director of a top 10 program (that I did not attend) who lamented the work ethic of residents these days. They implied it was an act of god to try to get them to stay in the OR past 3 PM and none of their graduates were looking for jobs that required significant call, even if the pay was good. They would rather take mommy track pay and mommy track hours than work for the big bucks.


I have no idea of how to judge the claims, but they had no reason to lie to me.

But those things are not a lack of professionalism. Just people who hold their life outside of work as a higher priority than work and money.

 

[Mman]

But those things are not a lack of professionalism. Just people who hold their life outside of work as a higher priority then work and money.

while I mostly agree, being angry about working more than 40 hours in a week as a resident would seem to be a serious lack of professionalism to me.

Also as a specialty, we need residency graduates that are going to lead the profession, not just punch a clock from 7-3 like a CRNA. Hospitals run 24/7/365 and we need anesthesiologists working every minute.

 

[DocVapor]

while I mostly agree, being angry about working more than 40 hours in a week as a resident would seem to be a serious lack of professionalism to me.

Also as a specialty, we need residency graduates that are going to lead the profession, not just punch a clock from 7-3 like a CRNA. Hospitals run 24/7/365 and we need anesthesiologists working every minute.

I agree with your first paragraph. But why do all that leading when our national societies keep selling us out? I'm more than willing to take the call, do the work, and lead, but when it's obvious none of that will do any good for us, why the hell would I put in the extra effort?

 

[abolt18]

while I mostly agree, being angry about working more than 40 hours in a week as a resident would seem to be a serious lack of professionalism to me.

Also as a specialty, we need residency graduates that are going to lead the profession, not just punch a clock from 7-3 like a CRNA. Hospitals run 24/7/365 and we need anesthesiologists working every minute.

I hear that. I was focusing more on the jobs that they were seeking out. If you're working under 60 hrs a week as a resident then you have nothing to complain about IMO.

But as far as choosing jobs, if people are seeking out jobs that won't consume their entire lives including nights and weekends, that's their prerogative. I don't see how that could be considered "unprofessional."

 

[dhb]

we need anesthesiologists working every minute.

No thanks, pass.

 

[MirrorTodd]

perhaps

I spoke within the last 12 months to a program director of a top 10 program (that I did not attend) who lamented the work ethic of residents these days. They implied it was an act of god to try to get them to stay in the OR past 3 PM and none of their graduates were looking for jobs that required significant call, even if the pay was good. They would rather take mommy track pay and mommy track hours than work for the big bucks.


I have no idea of how to judge the claims, but they had no reason to lie to me.

I'd argue that's a leadership issue and not a resident issue. We may complain to each other as residents, but you'd be hard pressed to find an Army or Navy anesthesia resident complying about having to stay past three here where I am training.

 

[gasdoc77]

perhaps

I spoke within the last 12 months to a program director of a top 10 program (that I did not attend) who lamented the work ethic of residents these days. They implied it was an act of god to try to get them to stay in the OR past 3 PM and none of their graduates were looking for jobs that required significant call, even if the pay was good. They would rather take mommy track pay and mommy track hours than work for the big bucks.


I have no idea of how to judge the claims, but they had no reason to lie to me.

Maybe the newer grads have figured out that even if the boomers aren't screwing them out of partnership opportunities, the elected boomers will print away the value of those extra big bucks earnings to pay for all the entitlement spending.

 

[vector2]

Did the acgme dumb down their requirements for case minimums? As you've said this doesn't seem like an osce centered solution.

It's not that the ACGME is lowering their requirements. It's that there's an understanding among multiple specialties that ACGME minimums are just a hilariously low threshold to hit. They're the equivalent of that silly "5 central lines and you're 'checked off'" policy that many hospitals have. Ultimately, the minimums in no way imply competency has been met.

Now, it wasn't a big deal when the vast majority of programs were based at tertiary academic centers. It's a breeze getting 50-100% more than the minimums in places like those. But what happens when you have an expansion of programs where the residents say struggle to meet minimums for double lumen tubes/thoracic cases? And if they do meet them it's community wedge biopsies or VATS for retained hemothoraces in healthy, skinny 20 yo trauma pts coming from the floor? These kind of programs - and believe me they do exist - do not put out residents who are competent enough to handle complicated cases.

That being said, there is a way for simulation or something OSCE-adjacent, in conjunction with the oral boards, to tease out which residents meet actually advanced competency vs those who've hit 20 b&b CABGs, 50 b&b c-secs etc and moved on.

 

[Mman]

I agree with your first paragraph. But why do all that leading when our national societies keep selling us out? I'm more than willing to take the call, do the work, and lead, but when it's obvious none of that will do any good for us, why the hell would I put in the extra effort?

your leadership is at the local hospital level. Being in charge of the OR that runs 24/7. You need actual anesthesiologists there doing the work and running the show.

The national society type stuff is irrelevant to your day to day work. When you say "it's obvious none of that will do any good for us", you could not be more wrong. It is precisely that local work and leadership that will prevent you from ever being replaced in your hospital. While it's nice as a specialty to be all in together in terms of lobbying and what not, some random podunk hospital in Wisconsin getting rid of their anesthesiologists is not going to impact my job because of the work I and my partners do every day.

It is precisely the docs that just show up and don't care and punch the clock that are the ones that will be replaced.

 

[coffeebythelake]

It's not that the ACGME is lowering their requirements. It's that there's an understanding among multiple specialties that ACGME minimums are just a hilariously low threshold to hit. They're the equivalent of that silly "5 central lines and you're 'checked off'" policy that many hospitals have. Ultimately, the minimums in no way imply competency has been met.

Now, it wasn't a big deal when the vast majority of programs were based at tertiary academic centers. It's a breeze getting 50-100% more than the minimums in places like those. But what happens when you have an expansion of programs where the residents say struggle to meet minimums for double lumen tubes/thoracic cases? And if they do meet them it's community wedge biopsies or VATS for retained hemothoraces in healthy, skinny 20 yo trauma pts coming from the floor? These kind of programs - and believe me they do exist - do not put out residents who are competent enough to handle complicated cases.

That being said, there is a way for simulation or something OSCE-adjacent, in conjunction with the oral boards, to tease out which residents meet actually advanced competency vs those who've hit 20 b&b CABGs, 50 b&b c-secs etc and moved on.

Seems like a pretty roundabout way of achieving this goal. Just mandate x number of procedures as prereq for board cert.

 

[vector2]

Seems like a pretty roundabout way of achieving this goal. Just mandate x number of procedures as prereq for board cert.

It's not just quantity, but quality- because my 40th normal EF b&b CABG doesn't really add much more to my 20th. At least not the same way a sick, redo CABG/double valve would. And I don't see the ACGME having the time or will to get that in depth with each specialty's certification process. If anything, they're moving in the other direction (the anesthesia trauma requirement turned into some vague "life threatening pathology" nonsense, etc)

 

[coffeebythelake]

It's not just quantity, but quality- because my 40th normal EF b&b CABG doesn't really add much more to my 20th. At least not the same way a sick, redo CABG/double valve would. And I don't see the ACGME having the time or will to get that in depth with each specialty's certification process. If anything, they're moving in the other direction (the anesthesia trauma requirement turned into some vague "life threatening pathology" nonsense, etc)

Still confused how an osce will somehow fix this problem

 

[vector2]

Still confused how an osce will somehow fix this problem

An OSCE is a type of a simulation. Regardless of what one thinks of simulations, there are in fact worthwhile ones which can accurately differentiate whether someone had adequate enough training to be capable of complex medical management.

Now while I doubt that the ABA OSCE will ever reach those heights of relevancy, the ABA's incompetence doesn't invalidate the underlying potential worth.

 

[coffeebythelake]

An OSCE is a type of a simulation. Regardless of what one thinks of simulations, there are in fact worthwhile ones which can accurately differentiate whether someone had adequate enough training to be capable of complex medical management.

Now while I doubt that the ABA OSCE will ever reach those heights of relevancy, the ABA's incompetence doesn't invalidate the underlying potential worth.

When ABA makes such a claim that OSCE will solve a training or knowledge deficit, the onus should be on them to show it actually has an outcome difference. Without that it is as others have said here - a pathetic money grab.

I think the same thing about MOCA (and USMLE Step 2 CS which we derisively called a foreign grad language proficiency test back in the day)

When there is such an asymmetric balance of power here it is basically extortion. ABA knows that board certification is essentially a requirement for anesthesiologists these days, and is tied to hiring, compensation, advancement, etc. This isn't 30 years ago when board certification was an exclusive club for a small subgroup of anesthesiologists.

 

[nimbus]

while I mostly agree, being angry about working more than 40 hours in a week as a resident would seem to be a serious lack of professionalism to me.

Also as a specialty, we need residency graduates that are going to lead the profession, not just punch a clock from 7-3 like a CRNA. Hospitals run 24/7/365 and we need anesthesiologists working every minute.

Maybe they don’t have the right incentive. We are eat what you kill with no buyin. Our new hires work hard, take extra call, pick up extra outpatient work on their postcall days, pick up extra OB shifts, and many are among our top earners. I wouldn’t expect them to do all that extra work if they were on salary or if they only got a small bump for the extra work. People respond to fair incentives.

 

[coffeebythelake]

while I mostly agree, being angry about working more than 40 hours in a week as a resident would seem to be a serious lack of professionalism to me.

Residents should understand that training is rigorous. I agree with your overall sentiment about this.

Also as a specialty, we need residency graduates that are going to lead the profession, not just punch a clock from 7-3 like a CRNA. Hospitals run 24/7/365 and we need anesthesiologists working every minute.

That's unfortunately how healthcare systems and many incentive systems exist for physicians today. Not just anesthesiologists.