Got my score back today. 35 (ca-2) up from 30 last year. Interesting because 30 was 44th percentile last year for CA1s and this year a 30 is 56th percentile. So I don't buy that magic number (33) logic.
Also I find it interesting the mean for CA2s is 72% and for CA1s only 75% .
I understand your reluctance to believe it, but I still believe it to be valid, at least to some degree. The standard for your passing or not passing has nothing to do with how others who are taking the exam do (perhaps to a small degree, it might; I will explain in a minute). In the years I have been following this process, the standard for passing has remained pretty stable and the pass rates have varied. In the late 90's, the pass rate plummeted as less competitive candidates were going through the process. They did not lower the bar to keep an 88-92% pass rate, and it dropped into the 60's for percent passing a couple of times (based on ABA's own data they released). The pass rate for orals dropped as low as 58%, IIRC. I have heard this preached for many years. The minimum standards for achieving board certification do not change just because the group you take it with is super smart or super bad. It should be stable so that you know exactly what you need to achieve to attain it. That is where the scaled score numbers help. A 33 is barely scraping by and numerous knowledge gaps may be exposed when the orals come around. A 40 or greater(as a CA-III) will have significantly fewer knowledge gaps and a much better chance of passing the orals than someone who got a 33 (as a CA-III) despite the fact that they both got a "pass" on the written.
Granted, these are all pretty new exams, so the ABA holds standard setting sessions where PD's from around the country come in, take the exam, and determine what an acceptable pass rate should be. They do this the first few years until they have what they feel to be a more stable exam. The pass rate is super high at first, because they feel that it should not unfairly punish those who did not have anyone's prior experience with which to base their studying on. As the exam becomes more mature and the reliability of it is better known, they have higher expectations and the pass rate is slowly normalized to be more in line with their usual pass rates. So, I think that is really the only way that it could matter how other people did. If the exam is new enough that they are still doing standard setting exercises, this could potentially impact it, I guess. I think they try to make up for that by setting the bar a bit lower on the first few administrations of a new exam. Therefore, I would expect that this summer's first administration of the new advanced exam will have a lower bar to account for the fact that it is new and the takers do not have past experience of others to base anything upon in their prep.
Just for clarity, the advanced exam will also contain some Basic topics, it will just be much more heavily weighted to advanced topics.
This is a time of great change in the process, so the utility of this information may be sketchy for a couple of years, but I think that the 33 scaled score will likely correlate with a barely passing score that leaves you exposed to possibly fail the orals.
Disclosure: I am in no way affiliated with the ABA and take no position on whether they are good or bad. I do, however, follow the process closely and try to stay well versed on the trends and changes that have occurred. Just trying to give a little historical perspective.
