Is Modern Science Flawed?

[aSagacious]

Advertisement - Members don't see this ad

Sorry for the sensationalist title, I just wanted to draw attention to a deserving topic that has been in the news recently.

After a few recent newsworthy scientific scandals (Climategate, CFS XRMV Virus, Flu vaccine leads to Autism, etc) folks have been looking closely at how these studies could go unchecked for so long.

In fact, the most recent issue of Science has been dedicated to raising awareness on the overall lack of independent reproduction of studies and the bias towards positive results.

Here's a great article in The New Yorker called "The Truth Wears Off." It discusses the propensity for journals to select for positive findings and how this relates to various conflicts of interest. A couple brief excerpts:

Sterling saw that if ninety-seven per cent of psychology studies were proving their hypotheses, either psychologists were extraordinarily lucky or they published only the outcomes of successful experiments. In recent years, publication bias has mostly been seen as a problem for clinical trials, since pharmaceutical companies are less interested in publishing results that aren't favorable. But it's becoming increasingly clear that publication bias also produces major distortions in fields without large corporate incentives, such as psychology and ecology.

He notes that nobody even tries to replicate most science papers—there are simply too many. (According to Nature, a third of all studies never even get cited, let alone repeated.)

When a large number of studies have been done on a single subject, the data should follow a pattern: studies with a large sample size should all cluster around a common value—the true result—whereas those with a smaller sample size should exhibit a random scattering, since they're subject to greater sampling error. This pattern gives the graph its name, since the distribution resembles a funnel.
The funnel graph visually captures the distortions of selective reporting. For instance, after Palmer plotted every study of fluctuating asymmetry, he noticed that the distribution of results with smaller sample sizes wasn't random at all but instead skewed heavily toward positive results.

Anyway, just thought I'd share.

(... here's where I'm supposed to say "thoughts?")

 

[Sephiroth]

The Journal of Negative Results doesn't get nearly enough respect.

www.jnrbm.com/

Spread the love

 

[kpcrew]

Well with the system the way it is, people have a huge incentive to publish rather than obtain data for its own sake. The only way that a researcher would check someone else's work is if they're performing research that builds on those results. Otherwise, why would you waste precious time and money trying to replicate someone else's work? It's not like NIH grants are plentiful these days. Also, if you spent years on a project and only got negative data, it feels like a waste. I totally understand why people would unconsciously select for expected data or even purposefully falsify data.

 

[CodeBlu]

Sorry for the sensationalist title, I just wanted to draw attention to a deserving topic that has been in the news recently.

After a few recent newsworthy scientific scandals (Climategate, CFS XRMV Virus, Flu vaccine leads to Autism, etc) folks have been looking closely at how these studies could go unchecked for so long.

In fact, the most recent issue of Science has been dedicated to raising awareness on the overall lack of independent reproduction of studies and the bias towards positive results.

Here's a great article in The New Yorker called "The Truth Wears Off." It discusses the propensity for journals to select for positive findings and how this relates to various conflicts of interest. A couple brief excerpts:







Anyway, just thought I'd share.

(... here's where I'm supposed to say "thoughts?")

This is like deja-vu... wait... Saggy are you in my body, posting as you?

 

[Roo2]

Sorry for the sensationalist title, I just wanted to draw attention to a deserving topic that has been in the news recently.

After a few recent newsworthy scientific scandals (Climategate, CFS XRMV Virus, Flu vaccine leads to Autism, etc) folks have been looking closely at how these studies could go unchecked for so long.

I'm not sure what point you're trying to make here. Climategate doesn't have any academic supporters except for the Heartland Institute, the MMR-autism link ended with Andrew Wakefield being kicked out of his institution, and the XRMV in the blood supply is being investigated out of a (very prudent) overabundance of caution. Which studies have "[went] unchecked for so long"?

edit: which high profile studies, that is. I'm well aware that a huge chunk of research is not validated; one has to look no further than the PDB for evidence of that.

 

[Drrrrrr. Celty]

Huh... I always thought that most psychologists and behavioral researchers spend the majority of their time trying to replicate and refute other peoples research findings... Maybe it's just the graduate students at my school who are always trying to disprove top end research findings.

 

[StoicJosher]

Money is the key factor here. Results - money.

There is a graduate student working at my lab doing horizontal gene transfer studies. In all probability, there is a good chance that all he is going to get is negative results. Publishing is just going to boil down to how well he can spin the results to fit his story. In the end, making a relevant paper out of negative results are hard and there is that.

 

[sibitrum]

The other problem with a lack of publication of negative results is that it makes it more difficult to figure out was has already been done/the current boundaries of our current knowledge.

 

[kpcrew]

The other problem with a lack of publication of negative results is that it makes it more difficult to figure out was has already been done/the current boundaries of our current knowledge.

Then again, those results might be because of poor experimental design, incomplete analysis, problems with the chemicals or cells, etc. There are many reasons for why an experiment would be unsuccessful and there are probably a fair number of failed experiments. The number of published articles is increasing and there's enough information to go through without looking at what people tried but didn't work. I agree with the basic premise though. It's a shame that there isn't a program that allows you to sift through articles and find exactly what you need. The search engines for scholarly articles do a passable job, but they have a lot of room for improvement.

 

[IMSingular]

" Even the law of gravity hasn't always been perfect at predicting real-world phenomena. (In one test, physicists measuring gravity by means of deep boreholes in the Nevada desert found a two-and-a-half-per-cent discrepancy between the theoretical predictions and the actual data.) Despite these findings, second-generation antipsychotics are still widely prescribed, and our model of the neutron hasn't changed. The law of gravity remains the same. " Jonah Lehrer in The Truth Wears Off.


I can't make any sense out the above.

Experiments don't prove the hypothesis/law; they only prove that the hypothesis/law cannot be proved wrong. One experimental investigation is not a compelling reason to abandon a law. Experiments themselves could have been flawed. Are we going to give up law of gravity because one experiment seems to have failed. On the other hand motion of perehelion on Mercury was a compelling experiment, and Newton's laws were relegated to be approximate and Einstein's law were found to be in confirmation of experiment; however this only indicates that we should abandon Newton's law, or rather consider them approximate, but not that Einstein laws are the ultimate truth.

I think the article is quite fuzzy. In any case title of thread, in my opinion, should be "Is the practice of modern science flawed".

Well scientists can make mistakes and some times mistakes chug along in the books for a long time. A simple thing like not applying conservation of momentum wasn't done properly in an analysis of one of the phenomenon and was repeated in books for decades from 1930-1957; many mathematically interesting and elegant theories by eminent scientits such as Edward Teller were developed but all was in vain once Lee and Yang came up with "right" theory and Mrs Wu analysed experiments properly.

 

[Mithril]

The search engines for scholarly articles do a passable job, but they have a lot of room for improvement.

Are you sure you know how to properly use a medical search engine? Something like OvidSP Medline is an extremely powerful tool for finding the articles you want.

 

[kduck]

I'm not sure what point you're trying to make here. Climategate doesn't have any academic supporters except for the Heartland Institute, the MMR-autism link ended with Andrew Wakefield being kicked out of his institution, and the XRMV in the blood supply is being investigated out of a (very prudent) overabundance of caution. Which studies have "[went] unchecked for so long"?

edit: which high profile studies, that is. I'm well aware that a huge chunk of research is not validated; one has to look no further than the PDB for evidence of that.

Going with what you said about lack of academic support for some of this "sensational" research, I think the contribution of the media is overlooked. In trying to appear "balanced," journalists will seek out opposing arguments on an issue, even if the vast majority of the scientific community is in agreement on one side (like the vaccine/autism link - almost all scientists know it isn't true, but you can find a few who still espouse this theory). Then, the journalist presents them both as equal sides of an argument, giving the public the illusion that something is under debate when it isn't.

This isn't totally the same as the issue of unsubstantiated research or publication bias, but it kind of addresses the issue of "scandals" in science that seem to have taken forever to be overturned.

 

[Miss Alyssa]

I'm not sure what point you're trying to make here. Climategate doesn't have any academic supporters except for the Heartland Institute, the MMR-autism link ended with Andrew Wakefield being kicked out of his institution, and the XRMV in the blood supply is being investigated out of a (very prudent) overabundance of caution. Which studies have "[went] unchecked for so long"?

edit: which high profile studies, that is. I'm well aware that a huge chunk of research is not validated; one has to look no further than the PDB for evidence of that.

That is true. Wakefield's stuff was thrown out because through the years it could not be replicated or substantiated.

 

[wholeheartedly]

I'll second the influence of the media. While there are some science writers with a solid background, other just don't have an understanding of good research. They just want an attention grabbing sound bite.

We spent the first few minutes of each lecture in my Epidemiology class going over news articles on studies. It was always interesting to go to the original study and compare the two. Frequently the article didn't even involve the hypothesis. Just whatever little tidbit would get a headline.

 

[scarshapedstar]

The NSF's budget is cut year after year. Next year it gets $6.86 billion. Might sound like a lot but that funds more-or-less the entirety of basic (i.e. not involving silicon wafers or boner pills) research in the US, which is also unsurprisingly at an all-time low with every passing year.

I could compare that figure to various other punchlines like the money we pay to farmers not to grow corn, but what's the point. You increase that number, you make it easier to get grants for important yet less attention-grabbing work. It has jack squat to do with "modern science" unless by "modern" you mean "in the era of budget cuts for their own sake".

Oh, and there are some nefarious carbon-industry types who really just like putting the brakes on science in general. Yes, it's short-sighted. No, they don't care.

 

[eHombre]

The NSF's budget is cut year after year. Next year it gets $6.86 billion. Might sound like a lot but that funds more-or-less the entirety of basic (i.e. not involving silicon wafers or boner pills) research in the US, which is also unsurprisingly at an all-time low with every passing year.

A propos the above post, here are the award rates for NIH RO1 awards. You'll notice that applicants have increased ~40% over the past decade while awardees have increased ~0%.

 

[IMSingular]

A propos the above post, here are the award rates for NIH RO1 awards. You'll notice that applicants have increased ~40% over the past decade while awardees have increased ~0%.

Is that, for funding rate, constant dollars or nominal dollars?

 

[eHombre]

Is that, for funding rate, constant dollars or nominal dollars?

It's the funding rate, as in, the percent of applicants who become awardees. No dollars involved in the calculation.

While RO1s do vary in dollar amounts (depends on the needs of the proposed project) they are pretty comparable in terms of prestige and tenure-making ability.

 

[JHU 2010]

in epi we call this phenomenon "publication bias"

and you know i love ya saggy, but i have to respond to your title.

Q: "Is Modern Science Flawed?"
A: No.

 

[Tots]

in epi we call this phenomenon "publication bias"

and you know i love ya saggy, but i have to respond to your title.

Q: "Is Modern Science Flawed?"
A: No.

Have you seen Ben Goldcare's TED talk? He presents a funny graph on publication bias in studies on publication bias. http://www.ted.com/talks/ben_goldacre_battling_bad_science.html

You can see a similar study/graph here about publication bias of positive results in studies that report publication bias(though it did not find publication bias as is suggested by Goldcare above): http://www.bmj.com/content/331/7514/433.full

 

[JHU 2010]

Have you seen Ben Goldcare's TED talk? He presents a funny graph on publication bias in studies on publication bias. http://www.ted.com/talks/ben_goldacre_battling_bad_science.html

You can see a similar study/graph here about publication bias of positive results in studies that report publication bias(though it did not find publication bias as is suggested by Goldcare above): http://www.bmj.com/content/331/7514/433.full

i liked that TED talk!

yeah, bias is a big deal, and we spend so much time trying to adjust for it and correct for it in statistical analyses and meta-analyses that we often forget that the best source of bias reduction is good study design

as for publication bias, our publication structure itself is flawed. in that TED talk he talks about intentional withholding of data, but what about unintentional withholding? ever try and get published with negative (null) results? good luck

so it's not science that is flawed. scientific, epidemiologic, and statistical rigor account for bias and address how to avoid it. the flaw here is correct application of this rigor and dedication to good study design, comprehensive data analysis, and honest data presentation

 

[aSagacious]

and you know i love ya saggy, but i have to respond to your title.

Q: "Is Modern Science Flawed?"
A: No.

Haha, well I don't believe the 'premise' suggested by my title either. I just figured that I'd throw something absurd in there to get more people to click on it.

 

[wanderer]

Most experiments have negative results. Do you really need to publish that (insert random catalyst) is ineffective in most reactions?
If negative results were published all the time you'd have journals cluttered with useless dead-end experiments.

 

[CoolWhipp]

Most experiments have negative results. Do you really need to publish that (insert random catalyst) is ineffective in most reactions?
If negative results were published all the time you'd have journals cluttered with useless dead-end experiments.

And on the flip side, how many experiments were conducted trying to do something that many others have already failed at? How many tens of millions of dollars wasted?

I could argue either way, though.

 

[IMSingular]

Haha, well I don't believe the 'premise' suggested by my title either. I just figured that I'd throw something absurd in there to get more people to click on it.

One of the attributes of trolling I suppose.😉

 

[Isoprop]

Experiments ARE repeated. People are looking to find exceptions to a model or test if a theory will work in all cases.

 

[wanderer]

There's at least one chemistry journal that requires that experiments be repeated by another group before publication. And of course if a paper is important it will undoubtedly be repeated by some other group down the line anyways.

 

[deleted393595]

A propos the above post, here are the award rates for NIH RO1 awards. You'll notice that applicants have increased ~40% over the past decade while awardees have increased ~0%.

Agreed. There's a separate chart that I saw in last month's Science or Nature or Some Other Journal that adjusted the funding of the NIH in respect to inflation. End result? Funding has largely remained stagnant, while the number of funding requests has shot up.

In fact, I think the shortage is so severe that they are thinking of or have already implemented funding caps for the biggest grant proposals so that they can spread funding around to help keep the smaller labs in business.