The Role of Reputation in U.S. News & World Report's Rankings of the Top 50 American Hospitals
Ashwini R. Sehgal, MD
For the past 20 years, the newsmagazine U.S. News & World Report has published annual rankings of the top 50 American hospitals in several specialties. These rankings generate widespread attention among the general public, health care providers, and policymakers. Other media outlets, such as newspapers, amplify the visibility of these rankings through their own stories about top-ranked hospitals. Hospital leaders generally believe that the U.S. News & World Report rankings are accurate and use them as marketing tools to attract patients (1, 2). In addition, political leaders and health policy researchers searching for models for health care reform often focus on U.S. News & World Report's top-ranked hospitals (3).
U.S. News & World Report's rankings combine 3 quality domains with approximately equal weighting: structure, process, and outcomes. Structure is the people and resources involved in providing health care, process refers to how care is delivered, and outcomes are the results of such care. Structure and outcomes were assessed by objective measures, including nurse staffing, patient volume, patient safety adverse events, and adjusted mortality. Process was assessed primarily by the single subjective measure of hospital reputation. The rankings have been criticized for emphasizing too strongly the subjective reputation of a few nationally prominent hospitals, but the role of reputation has not been quantified (1, 4).
The goal of this study is to quantify the role of reputation in determining the relative standings of the top 50 hospitals in each of the 12 specialties evaluated (cancer; diabetes and endocrine disorders; digestive disorders; ear, nose, and throat; geriatric care; gynecology; heart and heart surgery; kidney disorders; neurology and neurosurgery; orthopedics; respiratory disorders; and urology). This study particularly attempts to determine whether a high degree of variation in the subjective reputation of hospitals results in a composite ranking that is dominated by subjective reputation. Understanding the role of reputation may help users interpret the rankings more knowledgeably and may inform efforts to improve the assessment of hospital quality.
[-- snip --]
Discussion
By combining several subjective and objective measures, U.S. News & World Report's rankings appear to be a rigorous, complex, and multidimensional index of hospital quality. However, the relative standings of the top 50 hospitals largely reflect the subjective reputation of those hospitals. This subjective reputation does not already capture objective measures of hospital quality; in fact,
little relationship exists between reputation and objective measures in the top 50 hospitals. The validity of these findings is strengthened by their consistency across all 12 specialties and across multiple complementary analytic approaches.
The predominant role of reputation is caused by an extremely high variation in reputation score compared with objective quality measures among the 50 top-ranked hospitals in each specialty (Table 2). As a result, reputation score contributes disproportionately to variation in total U.S. News score and therefore to the relative standings of the top 50 hospitals. In addition, all 100 randomly selected unranked hospitals had reputation scores of 0 or that were minimal. These data suggest that hospitals lacking national recognition are unlikely to be highly ranked by U.S. News & World Report.
This study's findings have implications for users of the rankings and for efforts to improve the assessment of hospital quality. Because obtaining national data on process is difficult, U.S. News & World Report relies on reputation as a proxy measure (5). Because reputation score is determined by asking approximately 250 specialists to identify the 5 best hospitals in their specialty, only nationally recognized hospitals are likely to be named frequently. High rankings also may enhance reputation, which in turn sustains or enhances rankings in subsequent years.
Users should understand that the relative standings of U.S. News & World Report's top 50 hospitals largely indicate national reputation, not objective measures of hospital quality.
Because reputation can deteriorate quickly from a single negative event, even highly rated hospitals should be concerned about rankings based primarily on reputation. Moreover, reputation-based rankings may act as barriers to improving hospital quality, because even highly successful quality improvement programs are unlikely to enhance a hospital's ranking. Assessors of hospital quality should be wary of subjective measures and of combining measures with large and small degrees of variation within a single index. Otherwise, the high variation measure will dominate differences among hospitals even if all measures are weighted equally in calculating the index.
Previous investigations have focused on validating U.S. News & World Report's rankings by comparison with external measures. Highly ranked cardiology hospitals were found to have a lower adjusted 30-day mortality among elderly patients with acute myocardial infarction (9). However,
many of these hospitals performed poorly in providing evidence-based care for patients with myocardial infarction or heart failure (10). Performance on Medicare's core measures related to myocardial infarction, congestive heart failure, and community-acquired pneumonia was also frequently discordant with U.S. News & World Report's rankings (11). The analyses presented here examined a broader range of measures internal to U.S. News & World Report's ranking system and
found little relationship between rankings and objective quality measures for most specialties (Table 4).
This study's findings apply primarily to interpretations about the relative standings of U.S. News & World Report's 50 top-ranked hospitals in each specialty and not necessarily to the hundreds of unranked hospitals. This study focused on the top 50 hospitals for several reasons. First, only the top 50 hospitals in each specialty are publicized by U.S. News & World Report, other media outlets, and the hospitals themselves. Second, unranked hospitals do not receive numerical rankings, making it difficult to examine the role of reputation in determining rankings. Third, the data needed to examine unranked hospitals are not readily available. The total U.S. News scores and some, but not all, subjective and objective components of unranked hospitals are available on a Web site. However, the information must be accessed separately for each combination of hospital and specialty (a total of 1859 hospitals × 12 specialties = 22 308 Web pages) (7).
This study also did not address the validity of using reputation as a process measure; the representativeness of the physicians who rated hospital reputation; the accuracy of the objective measures; or the absence of other key factors, such as patient satisfaction. The study's findings may not apply to other hospital quality-rating systems, such as the Medicare Hospital Quality Compare in the United States or the National Health Service's Performance Ratings in the United Kingdom. Medicare provides quantitative data on several performance measures but does not create an overall quality measure, whereas the National Health Service uses a semiquantitative approach to categorize overall quality as excellent, good, fair, or weak (12, 13).
U.S. News & World Report's method report states that its rankings are "based largely on hard data" (5). A necessary step for this claim to be valid would be to dramatically reduce or eliminate the role of reputation. In addition, the data used to develop the rankings should be more readily available to review.
The current rankings fall short of being an evidence-based system that data-conscious consumers, value-based purchasers, and reform-minded policymakers can rely on for health care decisions (1, 11).
http://www.annals.org/content/152/8/521.full
.
Us state schoolers gotta represent and show the private kids how it's done outside the ivory tower... or something. 
