• Scariest Story Contest

    Now that it's getting close to Halloween, we're running a contest to hear your scariest stories! These can be scary stories that you've experienced or stories that you've heard and the story with the most reactions will win!

    JOIN CONTEST

Psychology to come undone?

perhaps11

Full Member
7+ Year Member
Jan 11, 2012
153
194
  1. Psychologist
    I just happened to come by a very interesting (and possibly disturbing) article posted on the popular website, RealClearScience, regarding new efforts in replicating psychological experiments published in prestigious psychological journals (e.g. Journal of Social and Personality Psychology) beginning with articles published in 2008. While I personally welcome the idea of increased replication of psychological studies to ensure validity and reliability, the article and its inspiration, the newly formed Reproducibility Project, seems to project familar doubts that psychology is a sound science. Indeed, the article is literally titled "Psychology to become undone?" and mentions the so-called File Drawer Effect several times. :rolleyes:

    Here is the article:http://chronicle.com/blogs/percolator/is-psychology-about-to-come-undone/29045

    What do you guys make of this article, The Reproducibility Project, and its (possible) impact on research in psychology and psychological science as a whole?
     

    Ollie123

    Full Member
    10+ Year Member
    Feb 19, 2007
    5,217
    2,542
    1. Psychologist
      Been a hot topic in my department the last couple weeks.

      The overarching idea is nothing new and I've been following discussion of it for years. Personally, I couldn't be happier with the fact that it is occurring though obviously the devil is in the details. Probably the most disheartening aspect of graduate school for me was realizing the relative sloppiness of most research, and the fact that doing "good science" (i.e. careful, conservative analytic approaches, rigorous procedures for checking data) not only takes more work it almost inevitably leads to more convoluted and or null findings, and "worse" manuscripts (worse in quotations to indicate the interpretive ambiguity). Some of the most successful folks I know have actively argued against doing things like checking for normality, outliers, etc.

      My suspicion is that some effects will replicate, some won't. Depending on the particular methodology, I'd estimate the overall will be moderate with some variance across research areas. Not to pick on social, but I suspect they will likely be in the worst shape based on what I've heard from folks in the field, with a number of INCREDIBLY sketchy practices being commonplace and likely more variability attributable to the sample by nature of the things they study (e.g. it seems reasonable that views of gender and sexuality would be affected by geographic location/political climate relative to say...neural processing of vertical vs. horizontal lines).

      The main concern I've heard is that it will make psychology look bad. That's a possibility, but one I can live with. I don't think we are much worse than other fields (if at all). At the very least, medical research is also largely a train wreck! If anything, the fact that we are the driving force behind such a movement might open new avenues for psychologists to pursue as an authority on the "Science of Science" so to speak. Now is not the time, but if I was at a point I was running my own lab I would likely be throwing myself full force into the project and other related studies.
       
      About the Ads

      Ollie123

      Full Member
      10+ Year Member
      Feb 19, 2007
      5,217
      2,542
      1. Psychologist
        I am definitely saying its a bad thing - at least ethically. Like I said, the disheartening thing is unless the system changes, its probably good for your career. For the record, I should have said "successful early career folks" (i.e. other grad students and young faculty). Have yet to have a senior person say that, though I'm sure many do.

        And yes, it basically boils down to the results looking better that way and the relative ambiguity about how to handle data. I also know several people who have admitted they check for outliers only when hypotheses don't come out...not when they do. I'm sure the prevalence of that is just astronomical.
         

        Psychadelic2012

        PhD Student
        Sep 17, 2011
        827
        14
        1. Psychology Student
          Gasp. Sigh. Belch. Ugh.

          Seriously, though, it is very disappointing. And it is, of course, directly related to the fact that publishing null findings is...well, non-existent. I've never heard it said explicitly, but it is implied that whatever gets you a significant finding is what needs to be done. "Fishing? No, we don't fish. Fishing is bad. But keep looking, we'll find something."
           

          AcronymAllergy

          Neuropsychologist
          Volunteer Staff
          10+ Year Member
          Verified Expert
          Silver Member
        • Jan 7, 2010
          8,333
          3,461
          1. Psychologist
            Seems like a really great direction to go in. The disparaging comments about our discipline was surprising. We're a useless field apparently. Sigh.

            Well, in response to the negative tone of the article (and probably some of the comments), Dr. Nosek posted the following as his own comment, so it looks like he agrees with others' disappointments at the snarkiness of the blog:

            Tom -

            I am disappointed with the negative tone of this post. Reproducibility ("checking your work") is a central pillar of science. Conducting a replication is not a threat to the integrity of the publishing scientist or of the original work. It is an ordinary practice to improve the confidence in scientific knowledge. The scientist ideal is not to "be right" it is to "get it right."

            Further a failure to replicate does NOT mean that the effect is false or that the original researcher did something wrong. It *might* mean that the effect is false, but there are many contributors to a lack of reproducibility, and all of them are important to understand.

            Most important, the Reproducibility Project is a project of psychologists about psychology. We are investigating ourselves. The sample of studies is just that - a sample. From that sample, we hope to learn something about the population from which we drew it. That population includes me and my collaborators. If the results suggest low reproducibility, then I take it to mean as much about me and my laboratory as it does about any article in the sample.

            Science is self-critical. If there are problems, scientists aim to identify them and do something about it. That is what I love about science, and why I am a psychologist.
             

            cara susanna

            Full Member
            10+ Year Member
            Feb 10, 2008
            6,470
            3,805
            Midwest
            1. Psychologist
              I'm reading the comments and, wow, I'd forgotten how much people hate and disrespect our field. Wasn't there just something published recently about how ALL scientific fields are having this problem?
               
              Last edited:

              AcronymAllergy

              Neuropsychologist
              Volunteer Staff
              10+ Year Member
              Verified Expert
              Silver Member
            • Jan 7, 2010
              8,333
              3,461
              1. Psychologist
                I'm reading the comments and, wow, I'd forgotten how much people hate and disrespect our field. Wasn't there just something published recently about how ALL scientific fields are having this problem?

                I honestly wouldn't place too much credence in the comments as reflective of society's views psychology. Anonymous comments on pretty much anything online, as we've all probably seen, more often than not fall into the excessively-negative and critical variety. Just look through some of the things people "say" in response to news stories on CNN.com, for example.

                I have a feeling that if the subject were sociology, economics, law, or history instead of psychology, the general gist of the comments would be the same. Although yes, psych does have a history of sometimes being described in a less-than-favorable light when compared to the natural/"hard" sciences.
                 

                IT514

                Neuropsychologist
                10+ Year Member
                Oct 14, 2007
                333
                30
                  Some of those comments made me laugh.

                  I remember having a conversation with a physics grad student who liked to sit in on psych thesis presentations. He remarked that all psych studies are weak because we can get away with calling a .6 correlation "high." My comment to him was, "So what you are saying is that we should study humans EXACTLY same way we study rocks and planets?" His reply was something to the effect of "yeah, and what would be the problem with that?" I guess he wasnt paying attention during the lecture on closed vs open physical systems.
                   

                  wigflip

                  Full Member
                  Oct 18, 2010
                  1,548
                  6
                    I'm reading the comments and, wow, I'd forgotten how much people hate and disrespect our field. Wasn't there just something published recently about how ALL scientific fields are having this problem?

                    That's true, but it's also a Chronicle article where the comments tend to be more educated. At least, from my experience anyway.

                    Sorry to say it, but I think this (hatin' on psych) is pretty commonplace in academia. When I started my current grad program, I wasn't aware of how negatively psych was regarded. To be fair, the humanities folks I've been exposed to seem to revile all the social sciences (especially where quantitative methods dominate), but I've heard profs and grad students in a variety of social science and interdisciplinary fields denigrate psych in particular.

                    Also to be fair, at least some psychology folk seem to be complicit in upholding the traditional disciplinary hierarchy (with psych nestled close to the STEM fields on top, other social sciences below, humanities at the bottom). I know my undergrad psych profs and TAs didn't exact gild the lily when it came to discussing their views of other social science disciplines.

                    Though I think there are concrete reasons for these cross-disciplinary pissing contests (epistemological, theoretical, methodological differences, as well as the reality of differential funding from universities), cross-disciplinary rivalry and hostility isn't collegial or productive. I'm interested in what other disciplines have to offer in terms of the types of questions they are uniquely suited to answer and their approach to doing so.
                     

                    roubs

                    Full Member
                    10+ Year Member
                    Sep 15, 2006
                    1,084
                    5
                    1. Psychologist
                      Before I read Dr. Nosek's comment I was formulating a reply along those lines, of course he put it better than I could have :)

                      Saying psych research will come undone is a bit dramatic.

                      And to be clear, realclearscience aggregates stories. It is a blog post and aren't we shocked a blog post is overselling its point?
                       
                      About the Ads
                      This thread is more than 9 years old.

                      Your message may be considered spam for the following reasons:

                      1. Your new thread title is very short, and likely is unhelpful.
                      2. Your reply is very short and likely does not add anything to the thread.
                      3. Your reply is very long and likely does not add anything to the thread.
                      4. It is very likely that it does not need any further discussion and thus bumping it serves no purpose.
                      5. Your message is mostly quotes or spoilers.
                      6. Your reply has occurred very quickly after a previous reply and likely does not add anything to the thread.
                      7. This thread is locked.