- Joined
- May 28, 2010
- Messages
- 1,433
- Reaction score
- 813
"Like other standardized tests, the MCAT exam is an imperfect measure of test takers’ achievement. Examinees’scores can be dampened by factors like fatigue, test anxiety, and less than optimal test room conditions or they can be boosted by recent exposure to some of the tested topics.
A test’s reliability is a reflection of the accuracy with which it measures the knowledge and skills it targets. Reliability estimates are used to calculate the standard error of measurement (SEM) of examinees’ test scores. The SEM is useful because it defines the size of the confidence intervals that surround the reported scores. Confidence intervals represent the range of test scores within which examinees’ true achievement levels probably lie. For MCAT total scores, the SEM is two points.
An examinee’s MCAT total score plus or minus two points defines the 68% confidence interval. Adding and subtracting two points to an MCAT total score of 23, for example, defines a confidence band that begins at 21 and goes to 25. This means that in 68% of cases the true score for an examinee with a reported score of 23 lays within the band that goes from 21 to 25. Reviewing applicants’ scores with the confidence bands in mind prevents over-interpretation of small differences in test scores. Figures 3 and 4 illustrate how confidence intervals can be used to interpret MCAT total scores. The reported score for each examinee is shown with a blue box. The 68% confidence interval around each examinee’s score is shown by the dashed lines in the figure.
In Figure 3, examinee A scored 23 and examinee B scored 26. The 68% confidence intervals around these scores overlap. The overlap between the two confidence intervals suggests that the two reported scores may not be meaningfully different from each other. Figure 4 shows a score of 21 for examinee C and a score of 28 for examinee D. The confidence intervals around their scores do not overlap, suggesting the two scores are more likely to be meaningfully different from each other (compared to examinees A and B)."
Can be found here.
https://www.aamc.org/students/download/267622/data/mcatstudentselectionguide.pdf
So a student with a 32 has a 68% confidence interval of 30-34 and a person with a 35 has a confidence interval of 32-37. Their score's CIs overlap so the difference between them is marginal. Same for a 27 vs. a 30. Each person has a 68% CI two points above and below their scores and have CI overlap with scores 3 points above and below.
Again, another reason why the MCAT isn't the sole metric used in the admissions process.
A test’s reliability is a reflection of the accuracy with which it measures the knowledge and skills it targets. Reliability estimates are used to calculate the standard error of measurement (SEM) of examinees’ test scores. The SEM is useful because it defines the size of the confidence intervals that surround the reported scores. Confidence intervals represent the range of test scores within which examinees’ true achievement levels probably lie. For MCAT total scores, the SEM is two points.
An examinee’s MCAT total score plus or minus two points defines the 68% confidence interval. Adding and subtracting two points to an MCAT total score of 23, for example, defines a confidence band that begins at 21 and goes to 25. This means that in 68% of cases the true score for an examinee with a reported score of 23 lays within the band that goes from 21 to 25. Reviewing applicants’ scores with the confidence bands in mind prevents over-interpretation of small differences in test scores. Figures 3 and 4 illustrate how confidence intervals can be used to interpret MCAT total scores. The reported score for each examinee is shown with a blue box. The 68% confidence interval around each examinee’s score is shown by the dashed lines in the figure.
In Figure 3, examinee A scored 23 and examinee B scored 26. The 68% confidence intervals around these scores overlap. The overlap between the two confidence intervals suggests that the two reported scores may not be meaningfully different from each other. Figure 4 shows a score of 21 for examinee C and a score of 28 for examinee D. The confidence intervals around their scores do not overlap, suggesting the two scores are more likely to be meaningfully different from each other (compared to examinees A and B)."
Can be found here.
https://www.aamc.org/students/download/267622/data/mcatstudentselectionguide.pdf
So a student with a 32 has a 68% confidence interval of 30-34 and a person with a 35 has a confidence interval of 32-37. Their score's CIs overlap so the difference between them is marginal. Same for a 27 vs. a 30. Each person has a 68% CI two points above and below their scores and have CI overlap with scores 3 points above and below.
Again, another reason why the MCAT isn't the sole metric used in the admissions process.
