I got an email from the NBOME discussing the COMSAE. Fairly interesting...
http://www.nbome.org/comsae.asp
Interesting excerpts
"
The scores of Phase 1 and Level 1 were similar, with Level 1 scores being slightly higher. The COMLEX Level 1 scores of examinees who took the timed COMSAE exam were, on average, 26 points higher than their Phase 1 scores. For examinees who took the untimed COMSAE, the mean difference was 10 points less. It was not clear why examinees who took the untimed version scored lower. Since it was each examinees own decision to choose the timing mode, it can be inferred that the reasons for taking a particular timing mode might be related to the score differences between the two timing modes.
Out of 1152 examinees, 129 (11.2%) failed Phase 1, and 120 (10.4%) failed Level 1. The failing rates on both exams were very close to the actual Level 1 failing rate in the 2008-2009 testing year. Table 2 shows that 83% of this examinee group passed both exams, and 4.7% failed both. Analysis shows that 6.5% failed Phase 1 but improved to pass Level 1 later. On the other hand, 5.7% passed Phase 1 initially but failed Level 1 later. The chi-square test of this cross-classification table was significant, suggesting that passing or failing Phase 1 was significantly correlated to the passing or failing status on Level 1 (chi-square value=153.9149 and p-value<.0001).
Most importantly, the average COMLEX Level 1 score was higher than the average COMSAE Phase 1 score for this group of examinees. No doubt, the benefits came from becoming more familiar with COMLEX Level 1 and from conducting a self-assessment by taking COMSAE Phase 1.
The results also suggest that the timed COMSAE probably has higher self-assessment value because the testing conditions of timed COMSAE are more similar to the real timing conditions of COMLEX.
On the other hand, like any other measurement tools, COMSAE cannot predict COMLEX in absolute terms. T
here were still 5.7% of examinees who passed Phase 1 but failed COMLEX. For any COMSAE examinee, especially those who narrowly passed COMSAE, it is important to understand that passing COMSAE does not guarantee passing COMLEX. Without extra effort, it is not guaranteed that the same performance level on COMSAE can be maintained for COMLEX. "