Psychology to come undone?

SDN Members don't see this ad. (About Ads)

I just happened to come by a very interesting (and possibly disturbing) article posted on the popular website, RealClearScience, regarding new efforts in replicating psychological experiments published in prestigious psychological journals (e.g. Journal of Social and Personality Psychology) beginning with articles published in 2008. While I personally welcome the idea of increased replication of psychological studies to ensure validity and reliability, the article and its inspiration, the newly formed Reproducibility Project, seems to project familar doubts that psychology is a sound science. Indeed, the article is literally titled "Psychology to become undone?" and mentions the so-called File Drawer Effect several times.

Here is the article:http://chronicle.com/blogs/percolato...e-undone/29045

What do you guys make of this article, The Reproducibility Project, and its (possible) impact on research in psychology and psychological science as a whole?

Been a hot topic in my department the last couple weeks.

The overarching idea is nothing new and I've been following discussion of it for years. Personally, I couldn't be happier with the fact that it is occurring though obviously the devil is in the details. Probably the most disheartening aspect of graduate school for me was realizing the relative sloppiness of most research, and the fact that doing "good science" (i.e. careful, conservative analytic approaches, rigorous procedures for checking data) not only takes more work it almost inevitably leads to more convoluted and or null findings, and "worse" manuscripts (worse in quotations to indicate the interpretive ambiguity). Some of the most successful folks I know have actively argued against doing things like checking for normality, outliers, etc.

My suspicion is that some effects will replicate, some won't. Depending on the particular methodology, I'd estimate the overall will be moderate with some variance across research areas. Not to pick on social, but I suspect they will likely be in the worst shape based on what I've heard from folks in the field, with a number of INCREDIBLY sketchy practices being commonplace and likely more variability attributable to the sample by nature of the things they study (e.g. it seems reasonable that views of gender and sexuality would be affected by geographic location/political climate relative to say...neural processing of vertical vs. horizontal lines).

The main concern I've heard is that it will make psychology look bad. That's a possibility, but one I can live with. I don't think we are much worse than other fields (if at all). At the very least, medical research is also largely a train wreck! If anything, the fact that we are the driving force behind such a movement might open new avenues for psychologists to pursue as an authority on the "Science of Science" so to speak. Now is not the time, but if I was at a point I was running my own lab I would likely be throwing myself full force into the project and other related studies.

Originally Posted by Ollie123

Some of the most successful folks I know have actively argued against doing things like checking for normality, outliers, etc.

Why is this? Is it because they find significance only when the data is raw? And you are saying this is a bad thing, right?

I am definitely saying its a bad thing - at least ethically. Like I said, the disheartening thing is unless the system changes, its probably good for your career. For the record, I should have said "successful early career folks" (i.e. other grad students and young faculty). Have yet to have a senior person say that, though I'm sure many do.

And yes, it basically boils down to the results looking better that way and the relative ambiguity about how to handle data. I also know several people who have admitted they check for outliers only when hypotheses don't come out...not when they do. I'm sure the prevalence of that is just astronomical.

Gasp. Sigh. Belch. Ugh.

Seriously, though, it is very disappointing. And it is, of course, directly related to the fact that publishing null findings is...well, non-existent. I've never heard it said explicitly, but it is implied that whatever gets you a significant finding is what needs to be done. "Fishing? No, we don't fish. Fishing is bad. But keep looking, we'll find something."

Seems like a really great direction to go in. The disparaging comments about our discipline was surprising. We're a useless field apparently. Sigh.

Originally Posted by Ya Ya

Seems like a really great direction to go in. The disparaging comments about our discipline was surprising. We're a useless field apparently. Sigh.

Well, in response to the negative tone of the article (and probably some of the comments), Dr. Nosek posted the following as his own comment, so it looks like he agrees with others' disappointments at the snarkiness of the blog:

Tom -

I am disappointed with the negative tone of this post. Reproducibility ("checking your work") is a central pillar of science. Conducting a replication is not a threat to the integrity of the publishing scientist or of the original work. It is an ordinary practice to improve the confidence in scientific knowledge. The scientist ideal is not to "be right" it is to "get it right."

Further a failure to replicate does NOT mean that the effect is false or that the original researcher did something wrong. It *might* mean that the effect is false, but there are many contributors to a lack of reproducibility, and all of them are important to understand.

Most important, the Reproducibility Project is a project of psychologists about psychology. We are investigating ourselves. The sample of studies is just that - a sample. From that sample, we hope to learn something about the population from which we drew it. That population includes me and my collaborators. If the results suggest low reproducibility, then I take it to mean as much about me and my laboratory as it does about any article in the sample.

Science is self-critical. If there are problems, scientists aim to identify them and do something about it. That is what I love about science, and why I am a psychologist.

I'm reading the comments and, wow, I'd forgotten how much people hate and disrespect our field. Wasn't there just something published recently about how ALL scientific fields are having this problem?

Originally Posted by cara susanna

I'm reading the comments and, wow, I'd forgotten how much people hate and disrespect our field. Wasn't there just something published recently about how ALL scientific fields are having this problem?

I honestly wouldn't place too much credence in the comments as reflective of society's views psychology. Anonymous comments on pretty much anything online, as we've all probably seen, more often than not fall into the excessively-negative and critical variety. Just look through some of the things people "say" in response to news stories on CNN.com, for example.

I have a feeling that if the subject were sociology, economics, law, or history instead of psychology, the general gist of the comments would be the same. Although yes, psych does have a history of sometimes being described in a less-than-favorable light when compared to the natural/"hard" sciences.

That's true, but it's also a Chronicle article where the comments tend to be more educated. At least, from my experience anyway.

Some of those comments made me laugh.

I remember having a conversation with a physics grad student who liked to sit in on psych thesis presentations. He remarked that all psych studies are weak because we can get away with calling a .6 correlation "high." My comment to him was, "So what you are saying is that we should study humans EXACTLY same way we study rocks and planets?" His reply was something to the effect of "yeah, and what would be the problem with that?" I guess he wasnt paying attention during the lecture on closed vs open physical systems.

Originally Posted by cara susanna

I'm reading the comments and, wow, I'd forgotten how much people hate and disrespect our field. Wasn't there just something published recently about how ALL scientific fields are having this problem?

Originally Posted by cara susanna

That's true, but it's also a Chronicle article where the comments tend to be more educated. At least, from my experience anyway.

Sorry to say it, but I think this (hatin' on psych) is pretty commonplace in academia. When I started my current grad program, I wasn't aware of how negatively psych was regarded. To be fair, the humanities folks I've been exposed to seem to revile all the social sciences (especially where quantitative methods dominate), but I've heard profs and grad students in a variety of social science and interdisciplinary fields denigrate psych in particular.

Also to be fair, at least some psychology folk seem to be complicit in upholding the traditional disciplinary hierarchy (with psych nestled close to the STEM fields on top, other social sciences below, humanities at the bottom). I know my undergrad psych profs and TAs didn't exact gild the lily when it came to discussing their views of other social science disciplines.

Though I think there are concrete reasons for these cross-disciplinary pissing contests (epistemological, theoretical, methodological differences, as well as the reality of differential funding from universities), cross-disciplinary rivalry and hostility isn't collegial or productive. I'm interested in what other disciplines have to offer in terms of the types of questions they are uniquely suited to answer and their approach to doing so.

Before I read Dr. Nosek's comment I was formulating a reply along those lines, of course he put it better than I could have

Saying psych research will come undone is a bit dramatic.

And to be clear, realclearscience aggregates stories. It is a blog post and aren't we shocked a blog post is overselling its point?