Thanks for this reply, UnderDoc. I am always looking for ways to hone my statistics skills. Let me just reiterate that before I took Step 1, I used to believe that it was better to fail and retake. I had no fear of actually failing, and neither has anyone that I have ever heard make that statement. Now that I'm older and wiser (i.e. passed Step 1 and 2 CK), I believe it is foolish for anyone to say it is better to fail. In fact, those who truly fear failing usually pray they pass on the first try regardless of their score. Anyway, on to the stats blog...
UnderDoc said:
While I agree that it's better to pass on the first attempt, Firion's logic is a bit askew here. Since it can be safely assumed that 100% of re-takers failed on their first attempt, a 36% fail-rate on the re-take is not too bad (ie, almost 2/3 went from failure to passing). Therefore, your risk of failure cannot be "elevated" as Firion asserts. In fact, statistically it's gone from 100% failure rate to 36% failure rate.
Point taken, but your point of view differs in that you are looking from the perspective of having already failed and assigning a p value of failure as 1.00. If I bet on red in 00 roulette my p value for "failure" is 0.55 whether I win or lose. Now it is impossible to mathematically predict a failure rate for Step 1, but it is experimentally 10%, or 0.1, for 1st time takers. Now as repeater, that experimental failure rate goes to 0.36...and that is DESPITE extra studying efforts and time off. So I believe that your logic is faulty because the actual change in p value for failure is +0.26 (increased
a priori failure risk) as opposed to -0.64 as you propose.
UnderDoc said:
Again, there are some logical gaps here. If students were to retake the exam without studying then it might be safe to assume, as Firion does, that it would be only mere chance (error) that would contribute to a score sufficiently high enough to pass. However, I'll bet that most people who fail their boards, regardless of the circumstances, invest some extra study time in an attempt to assure that it doesn't happen again. The fact that 74% people pass on the second go-round proves this point. Were this not the case you would expect only a very small minority of re-takers to pass - generally those with borderline scores who benefit from error in the testing instrument to lift them beyond the passing threshold.
Because of how the OP worded the question, we are assuming the tester failed by 1 point, or got a 181. If this hypothetical tester retested without studying or using any other method to enhance his/her knowledge, the expected pass rate would be slightly less than 50% (the mean would be centered at 181 with a SD of 6 according the NBME statistics). Without knowing the mean scores of the retakers, I don't know if the 74% 2nd try pass rate is statistically significant, but I will assume that it is. EVEN SO, there is absolutely no evidence that a retaker who passes will get a monster score or even has a good chance of beating the mean.
The second part of my post was to speak to whiners (such as I once was) who specifically say they would rather fail in order to have a shot at a monster score on their second try. One's score on Step 1 is a function of how well the preclinical years have "sunk in" and of how well a candidate takes tests in general, and both of these factors are very hard to change in the short term. Additionally, the vast majority of "failers" ARE within 1 SEM of passing, so it is not only possible but PROBABLE that many of the second try passes are due to chance alone and not due to a further accumulation of knowledge.