Feb 20-March 11: why all that time?

[JamesPhilly]

---

Members do not see this ad.

Hi all,

As I try to describe the match process to my non-medical friends/family, I find that they ask really good questions that I don't know the answer to. For example, we rank programs by Feb 20th and they do the same. So, why do we have to wait until March 11th for knowing whether we matched or not?

It can't be to 'double check' the list unless they use their hands 🙂 Computers today are far too powerful. For comparison, I coauthored a study that examined 20-something variables from the birth records of everyone born in PA from 1970-1985. Running SAS I cleaned the data and figured out some fairly complex calculations in about 60 seconds on a normal desktop computer, maybe add another 20 seconds to output graphs for our poster. What the NRMP does in terms of mathematics is far more simple and has less data points

Are programs allowed to fidget with their lists after Feb 20th? A professor told me this wasn't possible, but I'm not sure if she is really part of the match process. With my limited insight into the 'behind the scene' workings of the match, this seems like the most likely explanation.

Thanks!

 

[Sheldor]

Hi all,

As I try to describe the match process to my non-medical friends/family, I find that they ask really good questions that I don't know the answer to. For example, we rank programs by Feb 20th and they do the same. So, why do we have to wait until March 11th for knowing whether we matched or not?

It can't be to 'double check' the list unless they use their hands 🙂 Computers today are far too powerful. For comparison, I coauthored a study that examined 20-something variables from the birth records of everyone born in PA from 1970-1985. Running SAS I cleaned the data and figured out some fairly complex calculations in about 60 seconds on a normal desktop computer, maybe add another 20 seconds to output graphs for our poster. What the NRMP does in terms of mathematics is far more simple and has less data points

Are programs allowed to fidget with their lists after Feb 20th? A professor told me this wasn't possible, but I'm not sure if she is really part of the match process. With my limited insight into the 'behind the scene' workings of the match, this seems like the most likely explanation.

Thanks!

Nobody can mess with the lists after Feb. 20th, the applicants or programs. It takes this long because they do actually double check by hand the results. I forget where I saw it, but the actual algorithm only takes a couple minutes to run. Then they go through and check for errors twice, I believe.

Its just a QA process, and while I hate the wait, I'd much rather have a rigorous QA then find out I matched at one place then have it taken away from me due to an error.

 

[chessknt]

Nobody can mess with the lists after Feb. 20th, the applicants or programs. It takes this long because they do actually double check by hand the results. I forget where I saw it, but the actual algorithm only takes a couple minutes to run. Then they go through and check for errors twice, I believe.

Its just a QA process, and while I hate the wait, I'd much rather have a rigorous QA then find out I matched at one place then have it taken away from me due to an error.

This is the party line.

In reality it is a combination of tradition and demonstration of power IMO.

 

[Sheldor]

This is the party line.

In reality it is a combination of tradition and demonstration of power IMO.

Its only 12 working days. I think that to expect them to officially decide the fate of over thirty thousand people, asking for that many days doesn't seem unreasonable.

Hell, it takes that long or longer to get our step exam scores back and those are multiple choice on a computer. Believe me, I'm with you and this wait has been excruciating, but 12 days isn't that long.

 

[JamesPhilly]

This is the party line.

In reality it is a combination of tradition and demonstration of power IMO.

Does anyone else get the feeling as medical students that you're always being jerked around? I am continually disappointed by the amount of times 'tradition' is the reason behind silly or unreasonable practices in medicine, ie why we still use white coats/ties in the hospital, the long hours, why scrub nurses get to harass us. It's so demoralizing and pointless

I was joking when I said they checked the lists by hand...I am pretty sure that would be near impossible to do. I would be interested to see the source on that one, and perhaps a picture of the NRMP looking at our ROLs with magnifying glasses 🙂 That would be the only way I believe that explanation

I don't mean to mock you Sheldor, you are correct, having an accurate match process is important. However, I think 3 weeks is a little gratuitous given the simple task they have and the power of everyday computers. I think you're giving them (the NRMP) too much credit. Also, it would not take minutes to run the match algorithm, we are talking about seconds.

It would take a day or two to have multiple people double/triple check the excel file or SAS file or whatever...if residency programs really can't alter their ROL, then tradition would seem to be the best explanation for this very large time span

I appreciate your responses, good luck tomorrow!

 

[JamesPhilly]

Hell, it takes that long or longer to get our step exam scores back and those are multiple choice on a computer. Believe me, I'm with you and this wait has been excruciating, but 12 days isn't that long.

That's true, but with exams you have to worry about incorrect questions or answers, maybe you have to throw out a couple of questions, maybe you get challenged on one or two and have to consider that first. I wrote the NBOME about a crap question and got it thrown out.

But with the NRMP process, it's a lot more straight forward. Anyway, like I said I see what you're saying about wanting the process to be accurate, but 12 working days for a calculation that takes seconds seems more entrenched in tradition than anything

 

[IMvrach]

In exams they also reuse the questions. They could grade them on the spot, but then it would make it easier to pass the questions/answers to others.

With NRMP, there is no need to hand check an algorithm that was proven correct, verified and was not changed for years. Moreover, if they modify the software, say, to make it complete in 30 seconds instead of a minute (the match is DEFINED by the algorithm, they can't change it) and introduce a bug, more likely it will be caught not by their staffers but by the students/programs after they release the results.

That's true, but with exams you have to worry about incorrect questions or answers, maybe you have to throw out a couple of questions, maybe you get challenged on one or two and have to consider that first. I wrote the NBOME about a crap question and got it thrown out.

But with the NRMP process, it's a lot more straight forward. Anyway, like I said I see what you're saying about wanting the process to be accurate, but 12 working days for a calculation that takes seconds seems more entrenched in tradition than anything

 

[lackli]

I don't mind waiting 12 days if it prevents screw ups like for the urology match 2005. Then again it seems like from the report they can get match results out in a few days if they really put the pedal to the metal.

http://forums.studentdoctor.net/showthread.php?t=175816

Executive Summary of Investigative Report

At its scheduled February meeting in Baltimore, the AUA Board of Directors sought detailed explanation of the serious error encountered in the 2005 Urology Residency Match on January 24. The AUA President appointed a committee of the Board, chaired by Secretary Dr. Carl Olsson, to seek out the root causes of the mismatch, and to suggest corrective measures for the future.

The committee secured relevant AUA internal documents including e-mails and memos, interviewed staff responsible for the Match process, and reviewed the comments of Program Directors and residency candidates directly affected by the error. The committee reached the following findings and conclusions:

1. Shortly after the Match results were disseminated on January 24, a problem was first noted by several Program Directors who called attention to unexpected vacancies in filling their residency slots. The initial staff response was that the "Match process was fair and accurate" because it was based on the logic of past Matches, and had been "manually audited." However, it became clear by later that day that there was in fact a serious error.

2. The mismatch was caused not by a computer glitch but by human error. AUA staff misapplied the Match algorithm, giving primary importance to the programs' preference rather than to the candidates' preference. Both the original Match and the "manual audit" were conducted by junior staff, who lacked full understanding of the Match Guidelines.

3. Three major staff errors were identified:
* Senior staff responsible for the Match failed to follow standard IT procedures, which required testing and validation of the system and its results, before those results were released. "Red flags" were dismissed when they appeared.
* No one sufficiently knowledgeable about the residency programs reviewed the Match outcomes to judge their conformance with expected results.
* When the Match error was confirmed, senior staff did not respond with timely and appropriate apologies, or with explanations to those adversely affected by the results.

4. The mismatch was corrected within hours of first notice, and the corrected and verified Match results were communicated to all participants the same week (January 28). Unfortunately, because of a fax equipment problem, the candidates learned their results before the Program Directors were informed. Later apologies were sent to all concerned by AUA President Dr. Brendan Fox.

5. Importantly, the final Residency Match results were tested and accuracy was confirmed by an independent auditor.

Appropriate steps have been taken to ensure there will be no recurrence of the problems experienced this year. In the future, the Residency Match Review Process will include oversight by urologists from AUA Board of Directors and from the Society of Urology Chairpersons and Program Directors, an affiliate of the Society for University Urologists.

 

[IMvrach]

As you see their "auditors" didn't find the problem until the results were released. I also don't understand why not to let students learn the results same time as the programs? NRMP could just require not to call programs first 24 hours after releasing the results, even though I can't see what question can I ask that knowing my name 24 hours in advance will help the program to answer my question. It's all seems just a tradition.

I don't mind waiting 12 days if it prevents screw ups like for the urology match 2005. Then again it seems like from the report they can get match results out in a few days if they really put the pedal to the metal.

http://forums.studentdoctor.net/showthread.php?t=175816

Executive Summary of Investigative Report

At its scheduled February meeting in Baltimore, the AUA Board of Directors sought detailed explanation of the serious error encountered in the 2005 Urology Residency Match on January 24. The AUA President appointed a committee of the Board, chaired by Secretary Dr. Carl Olsson, to seek out the root causes of the mismatch, and to suggest corrective measures for the future.

The committee secured relevant AUA internal documents including e-mails and memos, interviewed staff responsible for the Match process, and reviewed the comments of Program Directors and residency candidates directly affected by the error. The committee reached the following findings and conclusions:

1. Shortly after the Match results were disseminated on January 24, a problem was first noted by several Program Directors who called attention to unexpected vacancies in filling their residency slots. The initial staff response was that the "Match process was fair and accurate" because it was based on the logic of past Matches, and had been "manually audited." However, it became clear by later that day that there was in fact a serious error.

2. The mismatch was caused not by a computer glitch but by human error. AUA staff misapplied the Match algorithm, giving primary importance to the programs' preference rather than to the candidates' preference. Both the original Match and the "manual audit" were conducted by junior staff, who lacked full understanding of the Match Guidelines.

3. Three major staff errors were identified:
* Senior staff responsible for the Match failed to follow standard IT procedures, which required testing and validation of the system and its results, before those results were released. "Red flags" were dismissed when they appeared.
* No one sufficiently knowledgeable about the residency programs reviewed the Match outcomes to judge their conformance with expected results.
* When the Match error was confirmed, senior staff did not respond with timely and appropriate apologies, or with explanations to those adversely affected by the results.

4. The mismatch was corrected within hours of first notice, and the corrected and verified Match results were communicated to all participants the same week (January 28). Unfortunately, because of a fax equipment problem, the candidates learned their results before the Program Directors were informed. Later apologies were sent to all concerned by AUA President Dr. Brendan Fox.

5. Importantly, the final Residency Match results were tested and accuracy was confirmed by an independent auditor.

Appropriate steps have been taken to ensure there will be no recurrence of the problems experienced this year. In the future, the Residency Match Review Process will include oversight by urologists from AUA Board of Directors and from the Society of Urology Chairpersons and Program Directors, an affiliate of the Society for University Urologists.

 

[GoSpursGo]

In the big scheme of things, finding out where you go one week earlier is not going to make a difference. Screwing up where everyone will be spending the next 3-7 years of their lives would--they do not hand check all 30,000 rank lists, but apparently a decent number of randomly selected rank lists to make sure nothing funky happened. While theoretically there should be no need to reverify that the algorithm works every year, nobody would want to go unmatched and spend forever wondering if there's some small chance that there was a glitch in the program that screwed up my life. On the whole, this is not a huge inconvenience.

Furthermore, what exactly would be the motive for drawing this process out any longer? You think they like fielding questions from neurotic med students every day from 2/20 all the way through a few days after the SOAP ends? I'm sure they want this process to be over as badly as we do so they can get back to what I'm sure is a much more relaxed job during the "off-season."

 

[IMvrach]

It is THEORETICALLY impossible to partially check the results of the algorithm. To verify even a single random candidate you MUST consider the lists of ALL other candidates and of ALL programs. So any meaningful verification is done using software, which is already available, not manually. For instance, they might have several different implementations of the same algorithm and check that their results are identical, or check stability - randomly seed the algorithm and see that it always converges to the same matching, all of these checks are likely done as part of the original run. And when they found the results were somehow screwed, they just rerun the algorithm and reissued the results.

I believe that the delay is partly a tradition and partly, not to be underestimated, to justify their full time jobs.

They do not hand check all 30,000 rank lists, but apparently a decent number of randomly selected rank lists to make sure nothing funky happened. While theoretically there should be no need to reverify that the algorithm works every year, nobody would want to go unmatched and spend forever wondering if there's some small chance that there was a glitch in the program that screwed up my life. On the whole, this is not a huge inconvenience.

Furthermore, what exactly would be the motive for drawing this process out any longer? You think they like fielding questions from neurotic med students every day from 2/20 all the way through a few days after the SOAP ends? I'm sure they want this process to be over as badly as we do so they can get back to what I'm sure is a much more relaxed job during the "off-season."

 

[Ogenki]

Nobody can mess with the lists after Feb. 20th, the applicants or programs. It takes this long because they do actually double check by hand the results. I forget where I saw it, but the actual algorithm only takes a couple minutes to run. Then they go through and check for errors twice, I believe.

Its just a QA process, and while I hate the wait, I'd much rather have a rigorous QA then find out I matched at one place then have it taken away from me due to an error.

They've made obvious mistakes before (last year, one radiology department matched more applicants than they had positions and because the process isn't open to truly independent auditors, there's no way to know how many mistakes they've made that weren't something so blatant. Most errors are also human errors, so a data entry mistake can screw you over (there've been people screwed over by programs putting in the wrong name/number on their rank list and the data entry for nrmp probably doesn't do as good of a job as Residency Coordinators and Program Directors.)

Personally I think its just whim. I mean why tell schools and programs 1 day ahead? Why is it Fridays when a couple years ago, it was Thursday. Why not Monday?

 

[GoSpursGo]

Personally I think its just whim. I mean why tell schools and programs 1 day ahead? Why is it Fridays when a couple years ago, it was Thursday. Why not Monday?

I don't think we're going to come to an agreement on the 2.5 weeks thing, but this is pretty easy--pretty much every school has a match day ceremony, so letting the schools know a day early is a logistical necessity. Someone has to print out all those letter and stuff them in the envelopes. Some students like this particular tradition, so why not. Similarly, the wait till Friday is to let all the SOAPers have as much time as possible to find a job and thus have SOMETHING to put in the envelope to participate in the match day.

aPD has actually said there isn't really a great reason for programs to know a day in advance; he guesses it's so that they can get all of the new interns' info together and be ready to contact them early Friday afternoon, but they could probably get the info on Friday and still do the exact same thing. *shrug*

 

[JamesPhilly]

hey do not hand check all 30,000 rank lists, but apparently a decent number of randomly selected rank lists to make sure nothing funky happened.

EDIT: it looks like someone already addressed that this is not mathematically possible, so see comments above

Furthermore, what exactly would be the motive for drawing this process out any longer?

That's what we've been discussing: it seems the answer is a mix of tradition, and a means to rationalize their full-time employment. It's tradition like how wearing an unwashed white coat in a hospital full of contagious disease for 6 months is acceptable as long as you roll the sleeves up. White coats and ties are traditional, even though they risk lives, and so they stay.

Let's also be honest and say that the NRMP has the easiest job on Earth. I can't help but hear NRMP statements in the voice of Micheal Scott. "We have to quadruple check the lists and that takes weeks". That's crap for reasons that you yourself stated. The (incredibly simple) match algorithm is important, and we do want it to be accurate, but there is no reasonable explanation as to why that takes weeks

But you're right, this isn't a huge inconvenience, however that's not my gripe. Like I said in the beginning of this thread, I have to explain how we practice medicine to my family, and when I do that, I realize just how in entrenched in tradition we have become and how stupid it all is. It's antithetical to the spirit of medicine, which is to practice the scientific method and focus on good outcomes. This match process is just another way that we disregard our main mission, not to mention I feel like I'm being 'punked for the millionith time

Does complaining here change anything? no. But does it make me feel a bit better? a little 🙂

 

[K31]

It is THEORETICALLY impossible to partially check the results of the algorithm. To verify even a single random candidate you MUST consider the lists of ALL other candidates and of ALL programs. So any meaningful verification is done using software, which is already available, not manually. For instance, they might have several different implementations of the same algorithm and check that their results are identical, or check stability - randomly seed the algorithm and see that it always converges to the same matching, all of these checks are likely done as part of the original run. And when they found the results were somehow screwed, they just rerun the algorithm and reissued the results.

I believe that the delay is partly a tradition and partly, not to be underestimated, to justify their full time jobs.

It would be prohibitively laborious to check the results of the algorithm by hand. But with a random sample of applicants, it is possible to do things like check they couldn't have matched further up their ROL--eg verify that none of the applicants matched at programs further up a certain applicant's list were ranked lower than that applicant on the program's ROL.

 

[IMvrach]

An applicant cannot match "further up", one either matches to the program or not. But in any case, since the algorithm defines the match, i.e. they can't just implement something "more fair" each year, all they can change is to write additional tests. And I am sure that all verifiable scenarios you can think of, have tests implemented years ago. They have had years to generate billions of ROL and check that the algorithm is verified by the tests. The ONLY error that can occur is a data entry error.

Trust me, they don't need to verify the algorithm itself and its fairness during these three weeks. The only thing I could think of, it might be the algorithm prints what it believes the ROLs are and they check manually that there was no errors in transmission of the ROLs between the algorithm and their site.

It would be prohibitively laborious to check the results of the algorithm by hand. But with a random sample of applicants, it is possible to do things like check they couldn't have matched further up their ROL--eg verify that none of the applicants matched at programs further up a certain applicant's list were ranked lower than that applicant on the program's ROL.

 

[cheruka]

I used to work in the sotware industry as a Sr. Engineer. Nope, you don't need that much time to run optimization software, even for 30,000 people in the match. And you can not check it by hand. If it is a new algorithm or software, I can understand but something you use repeatedly, it does not take that much time.