1. Dismiss Notice
  2. Download free Tapatalk for iPhone or Tapatalk for Android for your phone and follow the SDN forums with push notifications.
    Dismiss Notice
Dismiss Notice
Visit Interview Feedback to view and submit interview information.

Deconstructing the US News Rankings: statistics gurus click here

Discussion in 'Pre-Medical - MD' started by MacGyver, Apr 6, 2004.

  1. MacGyver

    MacGyver Membership Revoked
    Removed

    Joined:
    Aug 8, 2001
    Messages:
    3,759
    Likes Received:
    4
    Pre-emptive strike: Yes, I know I've got a lot of time on my hands to be worried about this stuff, so please dont waste everybody's time with a bunch of "you've got too much time on your hands" BS. Humor me on this.

    I have downloaded the US News data for 60 schools that they published. I'm trying to recreate their rankings in an Excel spreadsheet. There are a total of 119 schools included in the rankings, but since US News doesnt publish data on all 119 schools, thats going to be one source of error between my rankings and US News. How big of an error that introduces is something I dont know.

    This Excel spreadsheet includes both the data published by US News as well as a section of columns to the right side that has data generated by me, trying to simulate the US News ranking methodology.

    You can download the Excel spreadsheet here (click on link below, then click on "US News Rankings" link. Excel spreadsheet will come up. Then you can save the file to your computer using the "save as" option under "file" menu)

    http://macgyver25.freewebspace.com/USnews.htm

    Here's a review of the US News methodology:

    http://www.usnews.com/usnews/edu/grad/rankings/about/05med_meth.php

    I've tried to follow the steps they've laid out, but I'm doing something wrong, because I'm not getting the same rankings they are. I think most of it has to do with the logarithm transformation they speak about. I dont know if this is a simple log(x) or ln(x) function or if there is a scaling factor or a more complex "logarithmic" function they are using. I've thought about emailing US News, but I doubt they would release this trade secret.

    All you statistics/excel gurus, please download the sheet, check my numbers (right side of the worksheet shaded in blue) and tell me what I'm doing wrong.

    Here's a detailed breakdown:

    1) First, I took the research funding and applied a log(x) function to it. This is the step with the most uncertainty in my mind, because US News doesnt describe it with sufficient detail.

    1) I used the Excel STANDARDIZE function, along with AVERAGE and STDEVP functions to standardize each indicator. This STANDARDIZE function returns negative values for numbers that are far enough below the mean. I'm not sure if thats a problem or not. US News says that they "standardized indicators about their means" so I'm not sure if there's enough detail given there to reconstruct what they are doing

    2) After standardizing, I applied the weights given by US News.

    3) After weighting, I totaled all indicators into a raw score. My raw scores dont agree with US News. My ranking is as follows:

    Harvard
    JHU
    WUSTL
    UPenn
    Duke
    UCSF
    UMich
    Columbia
    UWashington
    Stanford
    Yale
    Cornell
    Baylor
    UCLA
    Mayo
    Vanderbilt
    UPittsburgh
    UCSD
    UTSW-Dallas
    Emory
    Case-Western
    UNC
    Northwestern
    UChicago
    Mount-Sinai

    This doesnt jive with US News, so something is wrong with my math. Download the worksheet, check it out, and let me know what I'm doing wrong.
     
  2. Note: SDN Members do not see this ad.

  3. canada

    canada Member
    7+ Year Member

    Joined:
    Feb 18, 2004
    Messages:
    91
    Likes Received:
    0
    can't dl it, need a username and password i think
     
  4. MacGyver

    MacGyver Membership Revoked
    Removed

    Joined:
    Aug 8, 2001
    Messages:
    3,759
    Likes Received:
    4
    Should be fixed now.

    1) Click on http://macgyver25.freewebspace.com/

    2) Click on link that says "US News Rankings"

    3) Go to "File" --> "Save As" to save the file to your computer
     
  5. canada

    canada Member
    7+ Year Member

    Joined:
    Feb 18, 2004
    Messages:
    91
    Likes Received:
    0
    not sure if this has an effect but on the very right, you list the assessment scores 0.2 each for a total of 0.4 but then for the selectivity, you let the subcomponents add up to 1.0 instead of the total 0.2.
     
  6. MacGyver

    MacGyver Membership Revoked
    Removed

    Joined:
    Aug 8, 2001
    Messages:
    3,759
    Likes Received:
    4
    The methodology says that the overall selectivity is weighted 0.2, OF that 0.2, 65% is MCAT, 30% is GPA, and 5% is acceptance rate.

    These are the weights I calculated:

    Peer review: 0.2
    Residency directors: 0.2
    Research money: 0.3
    Faculty: 0.1
    MCAT: 0.65*.2 = 0.13
    GPA: 0.30*.2 = 0.06
    Accept rate: 0.05*.2 = 0.01

    Those numbers should add up to 1.0, and as long as they do, then I think the weighting is correct.
     
  7. AlternateSome1

    AlternateSome1 Burnt Out
    Physician 10+ Year Member

    Joined:
    Aug 2, 2002
    Messages:
    975
    Likes Received:
    15
    Status:
    Attending Physician
    Ok, this is a guess after a pretty quick look at your stats.

    Rather than making research equal 30% of the total decision, you have multiplied the research points by .3 and just tacked those points onto the final raw score. I think you need to establish the highest in each category, mark that as 100% of that category (for instance, Harvard would have .3 points from research), then weight the others based on what percent they have compared to the top scoring school.

    ~AS1~
     
  8. MDEntropy

    MDEntropy Member
    7+ Year Member

    Joined:
    Mar 18, 2004
    Messages:
    131
    Likes Received:
    0
    Status:
    Attending Physician
    Slight twist to Alternate

    I think you need to find the mean and standard deviation for each category. Then you need to come up with a percentile rank for each school (standardization - excel can do this) within each category beforing applying the weight for that category.
     
  9. exmike

    exmike NOR * CAL
    10+ Year Member

    Joined:
    May 19, 2003
    Messages:
    4,206
    Likes Received:
    10
    Status:
    Fellow [Any Field]
    perhaps US News uses the MCAT and GPA to calculate the selectivity, then uses the selectivity as a single variable. that would probably have some effect.
     
  10. MacGyver

    MacGyver Membership Revoked
    Removed

    Joined:
    Aug 8, 2001
    Messages:
    3,759
    Likes Received:
    4
    I see what you are saying.

    If I'm understanding your setup right, if there was a school who had the highest score in EVERY category (e.g. research, MCAT, GPA, etc) they would have a score of 1.0.

    Therefore, Harvard would have a 0.3 score (perfect) in the research funding category, whereas Mayo with its 11 faculty/student ratio would have a 0.1 perfect score in the faculty resources category.

    I tried this idea, but the rankings are still off.

    check out this version of the rankings (version #2) here:

    http://macgyver25.freewebspace.com/USnews.htm
     
  11. MacGyver

    MacGyver Membership Revoked
    Removed

    Joined:
    Aug 8, 2001
    Messages:
    3,759
    Likes Received:
    4
    Good point. I did what you suggested, and now the top 10 schools match the official US news ranks. There are still lots of discrepancies among the other ranks though, so its not htere yet.

    apparently there are other things going on that dont match what US News does.
     
  12. AlternateSome1

    AlternateSome1 Burnt Out
    Physician 10+ Year Member

    Joined:
    Aug 2, 2002
    Messages:
    975
    Likes Received:
    15
    Status:
    Attending Physician
    Ok, two more suggestions. Try weighting the peer and residency director review by the 5 point scale rather than the highest within that scale. After that, I would try rounding to different variables at different points in the calculation. If they round to 2 or 3 variables at the pre-raw score combination, it could alter school placements by a position or two.

    ~AS1~
     
  13. AlternateSome1

    AlternateSome1 Burnt Out
    Physician 10+ Year Member

    Joined:
    Aug 2, 2002
    Messages:
    975
    Likes Received:
    15
    Status:
    Attending Physician
    A way to test how close your numbers are would be to convert your raw scores similar to the way the weighted sections were. For instance, Harvard can be set to 100, then calculate the percent of each and round to a two digit number. This can at least help tell how far away some schools are from where they should be, and it could explain some ties.

    ~AS1~

    edit: This number should come up close to the USNEWS Overall Score.
     
  14. Yogi Bear

    Yogi Bear 2K Member
    7+ Year Member

    Joined:
    Oct 11, 2001
    Messages:
    2,416
    Likes Received:
    5
    hey,

    i tried analyzing usnews methodology awhile back and i think what they end up doing may possibly be to take the average of a bunch of mini-ranks. (i.e. they'd rank mcat from 1-125, gpa 1-125, % admission from 1-125), etc. from these rankings, they'll do a weighted average to calculate the total score (w/ the lowest score being the highest ranked, and standardizing it to 100 points). [if this is the case, the position of each category matters, not the actual values...so harvard's 1 trillion funding only goes so far)..and its conceivable that hopkins/washu can become #1 if they have high enough mcat/gpa] you can try this by using the excel function: =rank(). (look it up under help).

    -one possibilty what ur numbers don't add up is that u don't have a list of all the schools. i.e. if it is indeed by a series of mini-ranks (1-125) for 10 categories and u only have the top 1-60 listed online, a school that's ranked 50 overall may have a research ranking's that's 100. that's why your number gets messed up.

    just a theory. try implementing it.
     
  15. liquid magma

    liquid magma Member
    10+ Year Member

    Joined:
    Aug 15, 2002
    Messages:
    46
    Likes Received:
    0
    If I were you, I would email US News directly and request an explanation. If those hosers didn't follow proper and/or published statiscial methods, then this issue needs to be addressed publicly. But at the very least, as part of journalistic integrity and for the sake of credibility, they are required to make public the specifics of their methodology.

    Very interesting...

    --LM
     
  16. Yogi Bear

    Yogi Bear 2K Member
    7+ Year Member

    Joined:
    Oct 11, 2001
    Messages:
    2,416
    Likes Received:
    5
    why do they factor in ratio of faculty:students as a component but use total nih funding rather than a ratio of nih$:students?
     
  17. MacGyver

    MacGyver Membership Revoked
    Removed

    Joined:
    Aug 8, 2001
    Messages:
    3,759
    Likes Received:
    4
    I've just about given up on this thing. I tried the mini-ranks idea, and this is what happened:

    1) The schools below Harvard got really close to the US News overall ranks

    2) however, the relative rankings were all screwed up.
     
  18. MacGyver

    MacGyver Membership Revoked
    Removed

    Joined:
    Aug 8, 2001
    Messages:
    3,759
    Likes Received:
    4
    I think there is something wrong with the way I'm standardizing the data. The reason I say this is because my ranks are still giving Harvard a huge edge over the other schools (100 vs 84 for nearest competitor, when the official ranks the discrepancy between Harvard and #2 is only 4 percentage points).

    Specifically, the research funding category seems suspect to me. I used a LN and a LOG function for it, and there is still a very large deviance between Harvard and the other schools. US News just says they used "a logarithmic function to transform the data." I'm not sure exactly what they mean by that.

    I'm using the Excel STANDARDIZE function which has 3 inputs:

    1) The value to be standardized
    2) The standard deviation of the array
    3) The average of the array

    This function uses a Z score to transform the data, and gives negative values (because the curve is centered around zero). The US News methodology says they "standardized indicators about their means"

    Is that the same thing as what I'm doing? Are negative values generated by the standardization process a problem?
     

Share This Page