Journal reading groups and harsh criticism?

[futureapppsy2]

---

Members do not see this ad.

So, I've been in a couple of journal reading groups--both formal and informal--and in quite a few of them, pretty much every article we read comes out looking fatally flawed, and these are articles in mid- to high-tier journals by well-respected people. I agree that criticism and critique is important, but honestly, going off of some of these groups, I'd start to get the impression that no one has published good or even decent research, ever.

Has anyone else had this experience? Is the tendency towards extremely harsh criticism a function of very close reading and discussion by several people?

 

[Ollie123]

What Jon said. Our faculty have routinely pointed this out to us as one of the major differences between what has affectionately been dubbed "First year grad student syndrome" even though its applied across the board to younger professionals.

One of the most interesting classes I've taken was taught by a very senior professor, who had us read articles from every field...molecular biology, physics, computer science, etc. as well as psychology. Our job was just to discuss but we were not allowed to critique. Only thing we could discuss was "What it really means" and how it might be applied to our research. I think that pushed it a bit too far since I think some methods discussion is quite useful, but it was a good experience.

I think it happens for several reasons. Its usually pretty easy for any reasonably competent person to critique methods. Every study has significant flaws...as an obvious example, there is always the trade-off between internal and external validity to slam. Laboratory study? Well what does it mean in the real world? Longitudinal survey? Well clearly the effects are due to a third unmeasured variable and not what you are interested in. Its also safe - its easy to pick on the methods and feel reasonably assured you are correct - building theories from flawed studies, etc. invites criticism from fellow group members.

I say this as someone who is at least as interested in statistics/methods as I am in addiction (my primary research area), was certainly guilty of all of the above as an early grad student, and still falls into that trap quite often🙂 Changing ingrained behaviors is a slow process.....

 

[futureapppsy2]

Thanks, guys. I've also noticed that this seems to calm down a bit when students come in with or get more research experience, as they learn that research is inevitably about trade-offs and that it is a lot easier to talk about a "perfect" study than to do one..

 

[cara susanna]

To me that's one of the flaws of the peer-review process: you have to find flaws, or else it's like you haven't done your job as a reviewer. Like Jon said, you can always find something that you think the authors should have done better. I've never seen any reviews that are like "Great study, no suggestions!"

 

[Pragma]

To me that's one of the flaws of the peer-review process: you have to find flaws, or else it's like you haven't done your job as a reviewer. Like Jon said, you can always find something that you think the authors should have done better. I've never seen any reviews that are like "Great study, no suggestions!"

It only happened once, but I did get one paper accepted with reviews like that 🙂

Notably, it wasn't a top tier journal.

I just reviewed a manuscript last week, and I tried to be mindful of these issues. I think as JS said, we have to make sure things aren't overstated and also be sure that there isn't a huge issue with the methods or stats. I err on the side of giving constructive criticism with realistic suggestions to improve the paper.

 

[bmedclinic]

I agree completely with Ollie. My area isnt addiction but I imagine it's the same regardless of your speciality. There's always trade offs and you cant always "get more data for a few months" or "add in this control". We're (typically) dealing with humans, and they're not always as good of participants as we'd like them to be.

 

[PsychPhDone]

I agree with the sentiment. We tend to tear stuff apart, but like to wrap up with a "what can we conclude from this study?" question. I think it's a valuable exercise in critical thinking as long as you can keep your attitude in check.

 

[Therapist4Chnge]

I agree with the sentiment. We tend to tear stuff apart, but like to wrap up with a "what can we conclude from this study?" question. I think it's a valuable exercise in critical thinking as long as you can keep your attitude in check.

Absolutely. I think critical analysis of current research is a very important part of the learning process. I think too many people take journal articles at face value, particularly once they get a few years out.