Relooking Research Ranking Methodology

[roboyce]

I've never been a big fan of the US News rankings, but I do understand that they often play an important role in the decision making process for alot of people. As I've visited more and more schools, I've come to realize that those rankings don't really reflect the quality of reasearch or the access of medical students to take part in studies.

US News weights the total NIH research money recieved 20%. I think that this overly favors the largely institutions and those with a network of affiliated hospitals. Granted, I understand that this is important, but I'm not sure it is as important as the funding received by each professor, which allows for a more equal comparison among schools. US News gives a 10% weighting to this, but it divides the total NIH funding by the total number of faculty...still not perfect.

I've done some other calculations which I think are more telling, although no single number can accurately reflect the spirit of any institution. I found the results to be interesting if nothing else. I hope this helps some of you as well. Take it all with a grain of salt...please. I don't want to start some showdown here.

Here's what I found for the schools to which I applied. If you want me to run the numbers on other schools, let me know. Some schools, however, did not supply all the required information. All numbers were from US News.

Research Funding Per NIH Grant

Stanford: $1,315,000 (I'm suspicious of this number)
UCSF: $469,000
Wake Forest: $452,000 (shocked)
Duke: $433,000
Harvard: $415,000
Columbia: $404,000
UPenn: $365,000
UNC: $357,000
Yale: $355,000
Brown: $337,000
Vanderbilt: $307,000
Dartmouth: $278,000

Research Funding Per Total Faculty - As Used by US News

Stanford: $332,000
UCSF: $262,000
UPenn: $239,000
Duke: $191,000
Yale: $190,000
Harvard: $171,000
UNC: $167,000
Vanderbilt: $159,00
Brown: $147,000
Wake Forest: $141,000
Columbia: $123,000
Dartmouth: $117,000

Research Funding Per Total Faculty Involved in NIH Research

Ok this one only counts the faculty that are doing NIH research rather than just the total faculty...a little more telling in my opinion. Not all schools report this.

Dartmouth: $522,000
UNC: $472,000
UPenn: $464,000
Columbia: $379,000
Brown: $342,000
Harvard: $338,000
Vanderbilt: $292,000
Yale: $288,000

 

[nosugrefneb]

Did you apply to UChicago?

From the website: "5th in the country in NIH funding per faculty member, 1st in the country in HHMI investigators per capita, and 5th in the country in National Academy of Sciences members per capita."

 

[Law2Doc]

Did you apply to UChicago?

From the website: "5th in the country in NIH funding per faculty member, 1st in the country in HHMI investigators per capita, and 5th in the country in National Academy of Sciences members per capita."

Agree. A number of schools which brag about their per faculty member funding aren't on your list at all. So if you are really looking for a ranking, I think you need to go outside of just those to which you applied -- as a number of yours probably fall off such a new list.

 

[roboyce]

My intention is not replace the ranking system, but just provide another means of looking at schools. That's why I only did it for the schools to which I applied. I've added UChicago. I will continue to update upon requests.

Research Funding Per NIH Grant

Stanford: $1,315,000 (I'm suspicious of this number)
UCSF: $469,000
Wake Forest: $452,000
Duke: $433,000
Harvard: $415,000
Columbia: $404,000
UPenn: $365,000
UNC: $357,000
Yale: $355,000
UChicago: $352,000
Brown: $337,000
Vanderbilt: $307,000
Dartmouth: $278,000

Research Funding Per Total Faculty - As Used by US News

Stanford: $332,000
UChicago: $263,000
UCSF: $262,000
UPenn: $239,000
Duke: $191,000
Yale: $190,000
Harvard: $171,000
UNC: $167,000
Vanderbilt: $159,00
Brown: $147,000
Wake Forest: $141,000
Columbia: $123,000
Dartmouth: $117,000

Research Funding Per Total Faculty Involved in NIH Research

Ok this one only counts the faculty that are doing NIH research rather than just the total faculty...a little more telling in my opinion. Not all schools report this.

Dartmouth: $522,000
UChicago: $486,000
UNC: $472,000
UPenn: $464,000
Columbia: $379,000
Brown: $342,000
Harvard: $338,000
Vanderbilt: $292,000
Yale: $288,000

 

[Fantasy Sports]

I dont think this method is very good at all. Look at Wake Forest, ranked ahead of Duke and Harvard.

The thing is, with your method, schools with a few faculty but a couple stars who get a lot of NIH funding look much better than they should be.

There is a lot to be said about having a ton of good researchers versus having a few stellar ones. I think your ranking biases against schools that have made a commitment to having a lot of professors doing a lot of different kinds of research in different fields.

 

[roboyce]

I dont think this method is very good at all. Look at Wake Forest, ranked ahead of Duke and Harvard.

The thing is, with your method, schools with a few faculty but a couple stars who get a lot of NIH funding look much better than they should be.

There is a lot to be said about having a ton of good researchers versus having a few stellar ones. I think your ranking biases against schools that have made a commitment to having a lot of professors doing a lot of different kinds of research in different fields.

You have some valid points. I would agree that there is something to be said for the research "Disney Land" that is Harvard and all of its associated hospitals. There is probably more opportunity there to find a professor working on something unique or of special interest that anywhere else in the world. In addition, you might get alot if interdisciplinary collaboration as well. However, to make this the sole indicator of research quality is misleading. It's like saying GM has the best production team because they sell the most cars. It's a matter of scale. Look at how many more primary investigators they have than any other school in the country.

Thus, I think you have to look at other factors as well. I noted above that I'm not trying replicate any sort of ranking system. I just wanted to provide another window through which I could evaluate the schools to which I applied. Plus, every time I went to a school, they would talk about how they were "Top 5" in some category for research based on some quantifiable factor. Out of that, I sat down with the numbers (I'm currently unemployed) and cranked a few things out.

Now, to address your concerns about a bias towards small schools, I think that is the exception not the rule. Yes, there are some suprises (Wake and Dartmouth), but if you look at all three of the categories that I provided, you'll see that those schools do not repeat as top performers. So while they may sneak into one of the groupings, they don't top the list all the time. Thus, if you look at all three of the groups I presented, plus the total NIH Funding received (the more traditional rankings) I think you see a general trend of which schools are consistently in the top 10 or whatever. All I did was to provide another means of looking at the schools. Also, if it had a general bias towards smaller schools, you would see Dartmouth and Brown topping each category. This is not the case.

Additionally, since I have to go by the numbers provided by US News, without having access to the pure data, I can't assess the validity of what each school is reporting. So maybe Wake is just a glitch, just like I have a sneaking feeling that Stanford may be as well.

So, as I said before, take it all with a grain of salt. This is not me establishing a definitive ranking system. Who am I to judge. It's a reference. Make of it what you want. Take from it what you want.

R

 

[Law2Doc]

I dont think this method is very good at all. Look at Wake Forest, ranked ahead of Duke and Harvard.

The thing is, with your method, schools with a few faculty but a couple stars who get a lot of NIH funding look much better than they should be.

There is a lot to be said about having a ton of good researchers versus having a few stellar ones. I think your ranking biases against schools that have made a commitment to having a lot of professors doing a lot of different kinds of research in different fields.

It's like sports teams. Teams with a lot of above average players tend to do well, as do teams with a couple of superstars. The research ranking contemplates that a school is better if it brings in more money. The other proposed ranking postulates that the school is better if the average faculty member brings in more money. But clearly we see that some of the schools with the most grant money don't have it spread equally and are relying far too much on a few heavy hitters, who at some point could leave. Either way, the notion is that schools bringing in more grant money are better. Not sure how that translates to things like quality of education, board scores, match lists, etc.