When the US News rankings come out next week, consider this op-ed that was printed in the New York Times last year by University of Chicago economist Austan Goolsbee:
How Rankings Rate
New York Times
April 12, 2004
By Austan Goolsbee
From best-selling book lists to Zagat dinner recommendations to David Letterman's top 10, it's clear that Americans love rankings. Little surprise, then, that the release this month of U.S. News & World Report's annual ratings of the nation's graduate schools has prospective students swarming newsstands for advice on where to go. The list has an undeniably huge impact. Applications typically soar at universities that get top marks; some state governments even use the rankings to help them decide how much money to give to their universities.
Critics argue that rankings like these try to quantify the unquantifiable - that variables used to measure quality are often flawed and incomplete. But the often overlooked, and perhaps more troubling, problem is that the average person who uses these rankings may not understand basic statistics. If he did, he would know that even if all the right variables were included, once he got past the top spots, the distinctions between schools (or restaurants or places to retire) are often meaningless.
Take a look at the sales ranking data at Amazon.com. I once needed a publication, a World Bank technical paper on the regulatory environment in Bulgaria. The demand for technical working papers on Bulgaria being what it is, the publication's Amazon sales rank was extremely low (about 2.5 million down on the list). Yet after I bought it, a most amazing thing happened. My one purchase moved this working paper past almost one million other books. This happened because once buyers get out of the best sellers -where the difference in sales can be enormous - almost everything else is basically tied. The differences in rank are statistically meaningless, and small blips cause big changes.
This same is true for the rankings of all sorts of things.
In the New York City Zagat guide, for example, a restaurant that raised its food rating by three points would pass 50 restaurants if it went from 25 to 28 (from "excellent" to "extraordinary"), but 439 establishments if it went from 19 to 22 (both considered "very good").
Then there are the education rankings. For the large mass of schools that rate in the middle of lists like U.S. News & World Report's, rankings are extremely sensitive to small blips. Any criteria included in a survey, like undergraduate grade point average or percentage of applicants accepted, will have random variations over time.
This is perfectly normal. These fluctuations, though, cause the rankings of schools in the middle to move all over the place. It's why schools at the top seldom move more than a place or two, but the Tippie College of Business at the University of Iowa can move up five spots one year and down 18 the next (which is what just happened).
This sensitivity to small changes also tends to encourage bad behavior among university administrators. Some, for example, no longer report foreign students' verbal scores on standardized admissions tests to get an overall better result, or have wait-listed candidates that are likely to turn their schools down so their university looks more selective. These things may raise measured performance a tiny bit, but tiny bits can mean a lot when everyone is clustered in the middle.
The lessons are pretty obvious: outside the very top, be careful about basing decisions on rankings data. The differences are often statistically meaningless. If possible, whether you're interested in schools or restaurants, look at the last several surveys, instead of just the most recent one, since random blips tend to average out over time. Certainly don't pick a Chinese restaurant tonight because its rating is 20 versus another's 19.
For those who really can't get enough, it's now possible to find rankings of the rankings. U.S. News & World Report may get lots of attention, but as of last week its "Ultimate College Directory" for 2004 had an Amazon sales rank of 22,305, worse than the Intercollegiate Studies Institute's "Choosing the Right College 2004" at 19,752.
How is a rankings addict to choose? Next time, perhaps, you would do well to consider an alternative: the informative (and surprisingly popular) "Cartoon Guide to Statistics." Its sales rank is 8,600.
--
Austan Goolsbee is a professor of economics at the University of Chicago Graduate School of Business.
)
