Staged Examinations for primary certification...? Really?

[sevoflurane]

So, my wife and I are pretty tight with an academic peds ansthesiologist. We meet up in Florida every year to decompress and have a good time. She is quite involved in medical student and resident education. A topic of conversation at the dinner table this year was an eye opener for me. It seems that the ABA in now moving towards "staged" examinations for primary certification.

Please read:

http://www.theaba.org/Home/TrainingPrograms

This is total news to me. I had no idea the ABA was doing this. Does it sound familiar? It reminds me of some of the MOCA requirements. Mind you, this takes time away from residency AND it costs money.

I need to dive a little more into it, but aparently, if you don't do well, it serves as grounds to keep you in residency for an additional 6 months (of course, this will follow you thoughout the rest of your career). Now you need to prepare for USMLE I, II, III and then immediately get ready for the "basic, advanced and then the applied examinations". The Basic examn I bellieve will occur after your step III examination. Both of those exams during residency.... and lets not forget the in-training examinations.

Hmmm.... not sure I like it. Or maybe I'm just missing something here. Residency is stressfull enough. Now you need to prepare for step III and the "basic" during your training + ITEs. Probably not a big deal, but just wondering why things are changing. I thought the writtens followed by the orals were a good way to achieve certification.

---

Members don't see this ad.

 

[sevoflurane]

http://www.theaba.org/pdf/Staged-Timeline.pdf

 

[Vaporized]

My program mandated we take step 3 during intern year (which I would have done anyways). The basic was new for this year, I was the first class to take it. It is during the first months of ca2 which is bad timing since the non ca1 residents are taking much more call.

 

[urge]

I think the purpose is to weed out the weak residents from year 1. In the past the weak residents have managed to finalize their residency with no real deterrents because they were always "rehabbed" after a bad in-training, yet to fail again on the next one until it is time for the board.

If you fail this one you are not moving along. I think it is a good idea.


Edited

 

[leaverus]

I think the purpose is to weed out the weak residents from year 1. In the past the weak residents have managed to make finalize their residency with no real deterrents because they were always "rehabbed" after a bad in-training, yet to fail again on the next one until it is time for the board.

If you fail this one you are not moving along. I think it is a good idea.

Then what the hell's the point of a residency program? No one is good at year one. If the program can't turn them into competent anesthesiologists then i would argue that the program is worthless and not the resident (given that the resident is otherwise putting forth the effort, of course). Aren't the residency programs getting paid $$$ from the government to TRAIN people? And i have no idea what your 2nd sentence is saying ("make finalize??")

 

[pgg]

Our CA2s were in the first group who just took the basic exam a couple months ago. They said it had a lot of step-1-ish material.

I'm still not clear exactly what the ABA is accomplishing here. Seems to me the AKTs and ITEs were more than adequate for identifying residents at risk for failing the written.

 

[Noyac]

I don't know were I stand exactly on all this. But I do know that there has got to be a better way to find out just how well a resident did during their training. LORs are never much help. Program directors are interested in placing the residents in practices, especially the bad ones so that they don't stick around as attendings or fellows. As a hiring physician, I want to know how well a resident did on exams as much as I want to know their clinical skill level.

 

[pgg]

I think the purpose is to weed out the weak residents from year 1.

[...]

If you fail this one you are not moving along. I think it is a good idea.

Residency programs as a whole are notoriously bad at making that uncomfortable early call to drop someone, put them on probation, or extend residency. I agree with leaverus this should be the domain of PDs but apparently as a group they suck at it.


We had a thread a few years ago looking at some study published by a program that showed ITE scores vs self-reported hours of study, plus subjective clinical evals. What was striking about it to me is that there was NO correlation between subjective grades (even the one specifically assessing knowledge!) and exam scores. Every attending thought every resident was above average, even the ones with single digit exam scores. It was a total participation-trophy-o-rama.

And, programs have a motive to push marginal people through ... residents are free labor, and failing to graduate everyone on time reflects badly on the program itself.

So given the inability (or call it multi factorial reluctance) of programs to grade weak residents as weak and intervene early, maybe a real deal, pass-or-you're-held-back exam early in residency is a good thing.

 

[urge]

Then what the hell's the point of a residency program? No one is good at year one. If the program can't turn them into competent anesthesiologists then i would argue that the program is worthless and not the resident (given that the resident is otherwise putting forth the effort, of course). Aren't the residency programs getting paid $$$ from the government to TRAIN people? And i have no idea what your 2nd sentence is saying ("make finalize??")

I disagree with you.

Most of the people failing the tests is due to lack of effort. The only thing a program can do to stimulate a lazy resident is threaten to fire them.

But, if you say a resident is putting all the effort in and is still failing, then I don't think that resident should be allowed finish. We are not helping anyone by allowing a subpar person practice independently.

We are not in kindergarten anymore.

 

[urge]

We had a thread a few years ago looking at some study published by a program that showed ITE scores vs self-reported hours of study, plus subjective clinical evals. What was striking about it to me is that there was NO correlation between subjective grades (even the one specifically assessing knowledge!) and exam scores. Every attending thought every resident was above average, even the ones with single digit exam scores. It was a total participation-trophy-o-rama.

Which study is that?

Most attendings give directions on how they want the cases managed. If a resident is amicable, responsible, and follows instructions, they will look like good residents even though they might have zero knowledge or understanding of what is going on. That's how most residents appear to be competent. They just cannot come up with a plan.

 

[Noyac]

Which study is that?

Most attendings give directions on how they want the cases managed. If a resident is amicable, responsible, and follows instructions, they will look like good residents even though they might have zero knowledge or understanding of what is going on. That's how most residents appear to be competent. They just cannot come up with a plan.

its the responsibility of the attendings to decipher this. Unfortunately there are attendings who were just like these residents and can't cut it out in PP. Therefore, they remain in their training program and never achieve anesthesia bliss. I think this number is small but we have all seen them. So it is the responsibility of the knowledgeable attendings to assess residents accurately and either bring them up to par or encourage them to try another specialty. Remember, if a resident doesn't make it in one specialty, they can always go to another one that suits them better. We need to be more cognizant of who we are allowing to practice on their own.

 

[pgg]

Which study is that?

There were three or four articles we talked about, as I recall.

My search-fu is failing right now, but one of the articles was from A&A in Feb 2005.

The Resident Application Process and Its Correlation to Future Performance as a Resident
http://journals.lww.com/anesthesia-..._Resident_Application_Process_and_Its.36.aspx

Figure 1 in this article makes my point best:

Second row, 3rd graph has ITE scores for this batch of residents, showing a wide range of objectively measured performance. The other graphs, including the 1st one on the top row ("knowledge") are attendings' subjective evaluations of residents. And everyone was judged to be slightly above average.

What this tells me is that at least this particular program absolutely SUCKED at grading its residents. No correlation between subjective evals and an objective exam. If that's the norm, then maybe the ABA is on to something with creating this basic exam to force remediation and residency extensions on lagging residents ... because programs appear to be reluctant or unable to do it.

Assuming, of course, that the basic exam actually tests clinically relevant knowledge.

 

[ArMed]

As one person of the 92% of residents that passed the inaugural BASIC exam, I can confidently conclude that it was administered solely as a fundraising effort on behalf of the ABA. I especially appreciated studying my a$$ off only to be told after-the-fact that we should not expect a score.

 

[Libertas]

to be fair, that study was designed to look at pre-match evaluation of medical students (grades, step scores) and correlate to post match resident evaluations, not exactly the same conclusions are you are drawing from it.

The aba basic had a lot of random, non clinically relevant questions on it. Much less basic anesthesia knowledge than say, the akt-1 or 6. How can they get away without asking a single question on how many minutes of oxygen are left in the cylinder and call it a basic anesthesia exam?! The madness.

 

[Mad Jack]

I can think of $ome rea$on$ $taged examinations make $en$e...

 

[twoliter]

The total cost for the Basic and Advanced exams is the same as the current cost of the "old" Written exam. So, in this case, maybe $$$ is less of a factor. I could see that changing in the future, though.

 

[BuzzPhreed]

As one person of the 92% of residents that passed the inaugural BASIC exam, I can confidently conclude that it was administered solely as a fundraising effort on behalf of the ABA. I especially appreciated studying my a$$ off only to be told after-the-fact that we should not expect a score.

You got something out of it. Whether you realize it or not.

 

[pgg]

I concede that I'm stretching the conclusions of that study, but I believe it's true: as a whole, programs (ie, PDs and attendings who eval residents) do a poor job with identification and early intervention of at-risk residents. Not just our specialty.

 

[pgg]

You got something out of it. Whether you realize it or not.

Yep, time spent reading isn't wasted, even if you were tricked into doing it.

 

[ArMed]

I am forever grateful to the ABA for affording me the opportunity to self-study to a degree I never thought possible. However, I doubt subsequent batches of BASIC examinees will take the test as seriously as the inaugural group of testee's did.
Where's the limit? Make MOCA every 5 years. Charge $2500 a test.
Where is the line between certification of proven skills and abuse of power?

 

[pgg]

You're a resident, you ought to be self studying to a degree you never thought possible.

 

[BuzzPhreed]

Where is the line between certification of proven skills and abuse of power?

Just keep an eye on how the cardiologists are trying to deal with this very issue... That will be the precedent...

 

[seinfeld]

I have stated in threads before that when you reach residency the focus on education should be to ensure learning not ensure a basic knowledge set. No patient, and for that matter I do not want, a doctor around me who knows 65% of the required knowledge do the job. I also dont want a colleague who scores 99% on exams but has no technical skills in the OR and no social skills with the surgeons/nurses/patients.

Education is a lifelong, evolving process. Not a binge and purge. The material on the written s was never rocket science, it could be found in any anesthesia textbook, If you read and do some practice questions you should easily pass. When i was a resident too many of my fellow residents did not read, actually felt as if getting a low score on the year 1 ITE was beneficial. Beneficial to whom? Not them or their patients!

I contend the ABA should really focus on noncumbersome but frequent educational tools, ie 10-20 hours per year of required combo ACE/SEE (old knowledge and new knowledge) where 100% correct answers are required. I dislike the term assessment as it implies that a single snapshot and doent allow for repetition which is the key to long term memory.

I went to my kids 1st grade parents night. I am amazed how much education has changed and it explains why my daughter enjoys school and is doing so well.

 

[loveumms]

Just so you know - there was no increase in price with the staged exams so it's not for "fundraising". The basic + advanced costs the same as the advanced alone did (it's now just broken into two charges).

I believe the goal is to test different subject areas. If you can't master the BASIC knowledge then you shouldn't be allowed to move on in training.

I also think the OSCEs will test clinical judgement skills (things that are very difficult to test on written exams). Several other countries have been doing OSCEs for a long time and I think the ABA realized this might be something to incorporate to prove a candidate is in fact a consultant. I'm not sure what the cost will be but I'm betting it will not increase significantly over what the orals cost now.

I'm not saying what they are doing is right or wrong. It's definitely something that is hard to really study and although I think lots of people think the ABA is out to get us - it's likey they just want to preserve the field (as best they can in today's medical landscape).

 

[Guillemot]

Im guessing that all this change in the certification process will have neither positive nor negative impact on the quality of care patients receive. In 100 years though they will revert to the old ways though as shuffling things around is someone's job and their salary depends upon it.

 

[aneftp]

The issue with extending residency for 6 months after 2 failed tests is not practical. Graduate medical center clocks start clicking once residency begins.

The ABA HAS NOT THOUGHT this through. Many programs simply do not have funding to extend residents education and still pay the residents.

This may not look like a huge issue. But it will be for those 2-3% who have trouble with standardi tests.

Residents will be kicked out after 2 failed attempts. In work horse programs where residents don't have time to study will suffer.

 

[pgg]

I bet this was driven by programs, not the ABA. It's hard to fire or hold back residents, and the risk of getting sued by a dismissed resident is real. I think a lot of PDs might be quietly glad to have a 3rd party assessment early in residency that would let them hold back a struggling resident (the ITE is specifically not supposed to be used for promotion purposes). You've got to figure the single-digit %ile scorers are probably struggling clinically too ... now the PDs can say it's the ABA's fault you're with us another 6 months.

And on the bright side for the resident who's struggling - maybe the mandatory rehab from failing a test that means something (unlike the AKT-0, -1, -6, or CA-1 ITE) will redirect a bad trajectory EARLY enough to matter.

This may not look like a huge issue. But it will be for those 2-3% who have trouble with standardi tests.

I'll probably get yelled at here for my insensitivity and arrogance by stating this opinion (it'll be the second time in the last couple weeks) but I think the "trouble with standardized tests" bit is a cop out.

I just don't buy the notion that REALLY poor scores on written exams at this level are a result of some learning disability. By the time you get to residency, you've spent a decade of life repeatedly taking difficult and competitive written exams. By now the people true #2 pencil allergies, whatever that means, have been weeded out. I'll buy "exam anxiety" as a legit reason to trip on the oral because it's a genuinely new/weird/intense experience, but not a written. At this point, low single-digit scores are a preparation deficiency, not an ability deficiency.

 

[sevoflurane]

I bet this was driven by programs, not the ABA. It's hard to fire or hold back residents, and the risk of getting sued by a dismissed resident is real. I think a lot of PDs might be quietly glad to have a 3rd party assessment early in residency that would let them hold back a struggling resident (the ITE is specifically not supposed to be used for promotion purposes). You've got to figure the single-digit %ile scorers are probably struggling clinically too ... now the PDs can say it's the ABA's fault you're with us another 6 months.

And on the bright side for the resident who's struggling - maybe the mandatory rehab from failing a test that means something (unlike the AKT-0, -1, -6, or CA-1 ITE) will redirect a bad trajectory EARLY enough to matter.




I'll probably get yelled at here for my insensitivity and arrogance by stating this opinion (it'll be the second time in the last couple weeks) but I think the "trouble with standardized tests" bit is a cop out.

I just don't buy the notion that REALLY poor scores on written exams at this level are a result of some learning disability. By the time you get to residency, you've spent a decade of life repeatedly taking difficult and competitive written exams. By now the people true #2 pencil allergies, whatever that means, have been weeded out. I'll buy "exam anxiety" as a legit reason to trip on the oral because it's a genuinely new/weird/intense experience, but not a written. At this point, low single-digit scores are a preparation deficiency, not an ability deficiency.

Pgg. Are you in an academic center now? Kinda sounds like it. Awesome!
Lucky residents.

 

[pgg]

Pgg. Are you in an academic center now? Kinda sounds like it. Awesome!
Lucky residents.

Yes, I moved back to where I trained. Enjoying it so far - it's nice to get back to practicing the full spectrum of anesthesia. Or nearly so, I'm not doing neonates or cardiac here. We have fellowship trained guys eager to jump on the preemie grenades and not enough CT volume to let the non-CT-trained people dabble there. And no trauma to speak of but that's A-OK with me.

So far my hardest adjustment has been the amazingly slow pace. Instead of 30 minutes, lap choles and c-sections take 1-2 hours. Wakeups are harder when the closing party (resident, intern, or med student) is a spur-of-the-moment decision ... eyes open tube out when the drapes go down is hard. And actually most frustrating of all, the pains I take to get same-day-surgery patients to PACU wide awake and fast tracked for discharge seem to be wasted because the nurses in the PACU don't seem to have any desire to push the patients through to phase 2 - it just means they get another patient that much sooner.

I did a 7 1/2 hour robot nephrectomy last week ... to add insult to injury at the end the surgeon joked about my CA1 resident taking 15 minutes to get the a-line, which he worked on while they did the cysto stent, so there wasn't even a delay. And never mind that we had a 40-minute delay up front because the room was set up backwards ... because his room booking listed the wrong operative side. You can't make this stuff up.

But so far so good.

 

[sevoflurane]

Yes, I moved back to where I trained. Enjoying it so far - it's nice to get back to practicing the full spectrum of anesthesia. Or nearly so, I'm not doing neonates or cardiac here. We have fellowship trained guys eager to jump on the preemie grenades and not enough CT volume to let the non-CT-trained people dabble there. And no trauma to speak of but that's A-OK with me.

So far my hardest adjustment has been the amazingly slow pace. Instead of 30 minutes, lap choles and c-sections take 1-2 hours. Wakeups are harder when the closing party (resident, intern, or med student) is a spur-of-the-moment decision ... eyes open tube out when the drapes go down is hard. And actually most frustrating of all, the pains I take to get same-day-surgery patients to PACU wide awake and fast tracked for discharge seem to be wasted because the nurses in the PACU don't seem to have any desire to push the patients through to phase 2 - it just means they get another patient that much sooner.

I did a 7 1/2 hour robot nephrectomy last week ... to add insult to injury at the end the surgeon joked about my CA1 resident taking 15 minutes to get the a-line, which he worked on while they did the cysto stent, so there wasn't even a delay. And never mind that we had a 40-minute delay up front because the room was set up backwards ... because his room booking listed the wrong operative side. You can't make this stuff up.

But so far so good.

Right on man. I bet it's quite the change from doing your own cases and I'm sure it takes some getting used to... but I bet you're molding abilities for the next generation is stellar... despite the slow wakeups.

 

[sevoflurane]

FWIW, we started getting med studs at our shop. I fully enjoy it. I actually received the "golden apple" award last year. More a joke than anything, but I'm digging teaching (no residents though).