Misreporting of research data

[LadyHalcyon]

---

Members don't see this ad.

A top Cornell food researcher has had 13 studies retracted. That’s a lot.

Sent from my SM-G950U using SDN mobile

 

[LadyHalcyon]

A top Cornell food researcher has had 13 studies retracted. That’s a lot.

Sent from my SM-G950U using SDN mobile

"A 2012 survey of 2,000 psychologists found p-hacking tactics were commonplace. Fifty percent admitted to only reporting studies that panned out (ignoring data that was inconclusive). Around 20 percent admitted to stopping data collection after they got the result they were hoping for. Most of the respondents thought their actions were defensible. Many thought p-hacking was a way to find the real signal in all the noise."


No bueno.

Sent from my SM-G950U using SDN mobile

 

[deleted343839]

That guy's day has been coming for a while now. Unfortunately for every person who is outed like this, someone else is still the darling of their institution and getting a promotion.

I think that increasing dependence on the project-based funding model makes it more tempting to do shoddy if not fraudulent work.

 

[Fan_of_Meehl]

"A 2012 survey of 2,000 psychologists found p-hacking tactics were commonplace. Fifty percent admitted to only reporting studies that panned out (ignoring data that was inconclusive). Around 20 percent admitted to stopping data collection after they got the result they were hoping for. Most of the respondents thought their actions were defensible. Many thought p-hacking was a way to find the real signal in all the noise."


No bueno.

Sent from my SM-G950U using SDN mobile

Unfortunately, I witnessed this kind of behavior as a grad student by other students (and tacitly sanctioned by the professor). The research assistant of the prof would plop down a big 'ole intercorrelation matrix, they'd note the 'significant' correlations, and then proceed to (post hoc) come up with 'hypotheses' (really, post hoc explanations) and then proceed to write the papers/posters as if they'd predicted the correlations from theory. It's one thing that really turned me off to academia as a career choice. I used to joke about them 'Bonferroni-ing around' with the data (my phrase 🙂 ) and 'harvesting asterisks' (a phrase borrowed from Paul Meehl). I later studied quite a bit in the philosophy of science proper and it bothered me even more as time went on. It also made me inherently skeptical of the presumed 'sanctity' of empirical research findings from the literature.

 

[PsyDr]

What an amateur. 80% of clinical trials data in China are fraudulent.

80% of China’s clinical trial data are fraudulent, investigation finds

 

[LadyHalcyon]

Unfortunately, I witnessed this kind of behavior as a grad student by other students (and tacitly sanctioned by the professor). The research assistant of the prof would plop down a big 'ole intercorrelation matrix, they'd note the 'significant' correlations, and then proceed to (post hoc) come up with 'hypotheses' (really, post hoc explanations) and then proceed to write the papers/posters as if they'd predicted the correlations from theory. It's one thing that really turned me off to academia as a career choice. I used to joke about them 'Bonferroni-ing around' with the data (my phrase 🙂 ) and 'harvesting asterisks' (a phrase borrowed from Paul Meehl). I later studied quite a bit in the philosophy of science proper and it bothered me even more as time went on. It also made me inherently skeptical of the presumed 'sanctity' of empirical research findings from the literature.

Wow. I hear it happens but I am honestly shocked to hear that.

Sent from my SM-G950U using SDN mobile

 

[Fan_of_Meehl]

Wow. I hear it happens but I am honestly shocked to hear that.

Sent from my SM-G950U using SDN mobile

To me--though appalling--it's not really surprising. People (and institutions, and organizations) actively choose (through their behaviors) every day which 'gods' (so to speak) they worship. When you have organizations that 'worship' 'successful' research (finding significant correlations), then that's what you'll get as behavioral output from those who work (and thrive) in those organizations. When you have organizations that (through their behaviors--not just their words) 'worship' and reward ethical behavior, then you get ethical behavior as an output from the members (and promoted members) of that organization. Unfortunately, I've observed over my lifetime a pretty universal drift for all organizations (including our own professional organization, APA) away from actually valuing ethical behavior (regardless of what is said by their PR people) and rather rewarding other behavior (publication outputs, profit, 'kissing-ass-upwards' in the power hierarchy [damn, this happens a lot at VA]). The results are not surprising to me.

 

[WisNeuro]

Hey, could be worse, at least we weren't using the wrong cancer cells for 50 years

The Dirty Little Secret of Cancer Research | DiscoverMagazine.com

Psychology seems to take the brunt of the "bad" research reporting when, if you look at the numbers, other disciplines are just as bad, if not worse. And, some of those other disciplines bad research actually leads to poorer patient outcomes or possibly death. Heck, some entities (pharma) do bad research out in the open, using inappropriate analyses, and sell billions of dollars in drugs that are essentially placebos that give you diarrhea (donepezil). While I'm all about cleaning up the problems in our own house, it'd be refreshing to have the problem be acknowledged for being as widespread as it is elsewhere, too.

 

[Ollie123]

I think that increasing dependence on the project-based funding model makes it more tempting to do shoddy if not fraudulent work.

This. x100000. Ideas have been tossed around for doing "lab based" funding, which I think would solve a lot of issues if it was done right, though I'm not convinced the present group would do so.

I will also say that while I am absolutely an advocate for improving what we do, I worry sooooo much of the "compliance" stuff is misguided. Preregistration helps for clinical trials and small lab-based things, but has the potential to kill innovation in some areas where it is just untenable (i.e. large epi studies that might generate 400-500 discrete papers). Open data assumes the data posted is accurate. My biggest concerns are all the errors that go into collecting the data in the first place, which I feel like isn't a part of the conversation. Mostly, my concern is who pays for all these things. I'm happy to pre-register my trials, post my datasets and share my code. I cannot and will not dedicate my time to doing so unless you agree to reduce my output expectations accordingly, especially given I'm also now expected to draft and sign memos every time someone misses an item on a questionnaire (this is seriously a thing we have to do here). You want more done, do it/pay for it yourself or STFU.

I would be hard pressed to design a better system for encouraging shoddy/fraudulent work than the current one. And we seem so focused on missing the forest for the trees and applying bandaid solutions that ignore the underlying cause. Support your faculty and I think you'll find 99.9% of them want to do the right thing and will move heaven and earth to do better work. If you make it a game, teach us that "ethical" behavior as happily agreeing to fill out all forms in triplicate and that anyone whose project fails is out of a job....you'll get an increasing number of people doing things like this. And no "open science" movement is going to stop it, its just going to change how people go about doing it. Because psychology. Something I thought we were supposed to be good at in this field.

 

[WisNeuro]

This. x100000. Ideas have been tossed around for doing "lab based" funding, which I think would solve a lot of issues if it was done right, though I'm not convinced the present group would do so.

I will also say that while I am absolutely an advocate for improving what we do, I worry sooooo much of the "compliance" stuff is misguided. Preregistration helps for clinical trials and small lab-based things, but has the potential to kill innovation in some areas where it is just untenable (i.e. large epi studies that might generate 400-500 discrete papers). Open data assumes the data posted is accurate. My biggest concerns are all the errors that go into collecting the data in the first place, which I feel like isn't a part of the conversation. Mostly, my concern is who pays for all these things. I'm happy to pre-register my trials, post my datasets and share my code. I cannot and will not dedicate my time to doing so unless you agree to reduce my output expectations accordingly, especially given I'm also now expected to draft and sign memos every time someone misses an item on a questionnaire (this is seriously a thing we have to do here). You want more done, do it/pay for it yourself or STFU.

I agree somewhat. I actually like the idea of pre-registration for most things. But I also think post-hoc analyses are important for exploration and innovation. But, post-hoc analyses should be clearly labeled as such. Far too often someone just p-hacks a large dataset and writes up "positive" findings as if that's what they set out to do in the first place. The data is still important in some contexts, but it should just be taken in the context of the analysis, which should then guide future confirmatory studies.

 

[Fan_of_Meehl]

I agree somewhat. I actually like the idea of pre-registration for most things. But I also think post-hoc analyses are important for exploration and innovation. But, post-hoc analyses should be clearly labeled as such. Far too often someone just p-hacks a large dataset and writes up "positive" findings as if that's what they set out to do in the first place. The data is still important in some contexts, but it should just be taken in the context of the analysis, which should then guide future confirmatory studies.

Such an important distinction between: a) the context of discovery/exploration vs. b) the context of hypothesis testing or confirmation/ disconfirmation.

 

[Ollie123]

I agree somewhat. I actually like the idea of pre-registration for most things. But I also think post-hoc analyses are important for exploration and innovation. But, post-hoc analyses should be clearly labeled as such. Far too often someone just p-hacks a large dataset and writes up "positive" findings as if that's what they set out to do in the first place. The data is still important in some contexts, but it should just be taken in the context of the analysis, which should then guide future confirmatory studies.

Pre-registration certainly has a role to play - I've become quite jaded about the state of academia so hope I didn't come across as completely diminishing its importance. I do think it is the proverbial bandaid on the broken limb. Some random related thoughts:
1) Time (see above)
2) The absurdity of the idea that we CAN know pre-specify everything we will do. This came out of clinical trials that have "duh" outcome measures with reasonably known distributions (i.e. dead vs. not dead). It works fairly well there. Simple social psych vignette studies with 1-2 outcomes...makes sense. The kind of stuff I and most of my colleagues are doing? I have no earthly idea what the spatial distribution of GPS contact with certain environmental features or what the distribution of neural connectivity indices on a novel MRI task will be. I can't specify every nuance of my analytic plan because we quite literally may have to invent new analyses. Currently pre-registration does an extremely poor job of addressing these things. So its become a game of trying to write things vaguely to give myself the freedom to make well-informed decisions later.
3) We seem locked into pre-registration without acknowledging the multitude of other well-established means of ethically conducting these types of analysis. Machine learning is literally an entire field dedicated to it. Cross-validation techniques, etc. - tons of tools exist that we don't use. It won't work in all cases, but I think we're so focused on shoving everything into a clinical trials framework that we aren't considering other valid options.
4) Why is post-hoc analysis a badge of shame? Its a tool, its not inherently evil. Yes, label it as post-hoc. Yet when you are reviewing for some crummy IF=2 specialty journal, don't recommend rejecting something because the authors openly said it was post-hoc and mention the need for replication in the limitations section.

 

[WisNeuro]

Pre-registration certainly has a role to play - I've become quite jaded about the state of academia so hope I didn't come across as completely diminishing its importance. I do think it is the proverbial bandaid on the broken limb. Some random related thoughts:

4) Why is post-hoc analysis a badge of shame? Its a tool, its not inherently evil. Yes, label it as post-hoc. Yet when you are reviewing for some crummy IF=2 specialty journal, don't recommend rejecting something because the authors openly said it was post-hoc and mention the need for replication in the limitations section.

I don't think it's a badge of shame at all, I just think it should be explicitly stated. Otherwise, people who are either intentionally p-hacking, or just ignorantly p-hacking because they learned how to do "research" from *****s, misinterpret and write up their study in a misleading way to the future reader of such research. I think post-hoc analyses serve very important functions, when used the right way. Unfortunately, they are not a good deal of the time. I imagine almost all of us who have engaged in any serious type of research have seen the p-hacking and misuse.

Plenty of things we need to fix to get around this. Like the publishing of well designed studies that do not find the desired effect, which is very important in its own right. If people saw those for the importance of what they were, they may be less likely to scour the data for any p<.05 that they can find to rework what they set out to write in the first place.

 

[LadyHalcyon]

Pre-registration certainly has a role to play - I've become quite jaded about the state of academia so hope I didn't come across as completely diminishing its importance. I do think it is the proverbial bandaid on the broken limb. Some random related thoughts:
1) Time (see above)
2) The absurdity of the idea that we CAN know pre-specify everything we will do. This came out of clinical trials that have "duh" outcome measures with reasonably known distributions (i.e. dead vs. not dead). It works fairly well there. Simple social psych vignette studies with 1-2 outcomes...makes sense. The kind of stuff I and most of my colleagues are doing? I have no earthly idea what the spatial distribution of GPS contact with certain environmental features or what the distribution of neural connectivity indices on a novel MRI task will be. I can't specify every nuance of my analytic plan because we quite literally may have to invent new analyses. Currently pre-registration does an extremely poor job of addressing these things. So its become a game of trying to write things vaguely to give myself the freedom to make well-informed decisions later.
3) We seem locked into pre-registration without acknowledging the multitude of other well-established means of ethically conducting these types of analysis. Machine learning is literally an entire field dedicated to it. Cross-validation techniques, etc. - tons of tools exist that we don't use. It won't work in all cases, but I think we're so focused on shoving everything into a clinical trials framework that we aren't considering other valid options.
4) Why is post-hoc analysis a badge of shame? Its a tool, its not inherently evil. Yes, label it as post-hoc. Yet when you are reviewing for some crummy IF=2 specialty journal, don't recommend rejecting something because the authors openly said it was post-hoc and mention the need for replication in the limitations section.

You sound a lot like my old professor who I spoke with the other day. He is very jaded by academia and basically said he just wants to tune out, collect a paycheck, and do hobbies he enjoys. I'm not from a PhD program but I'm from a small psyd program. What he basically said is because the psyd program doesn't make enough money for the school the faculty are expected to do it all for less money. This obviously includes classes, research, charing dissertations, publishing, etc. The chair of our department doesn't want to increase the number of students we accept due to the integrity of the program but the university wants more money. I noticed a similar problem at the university associated with my internship. I never wanted to go into academia but what I've learned in the past couple of years has made pay attention to job satisfaction for academic psychologists.

Sent from my SM-G950U using SDN mobile

 

[ClinicalABA]

I'm reading thins with interest, having spent most of my career working (and researching/occasionally publishing) in a field (ABA) that is almost void of inferential statistics. While single-case, inductive research is not necessarily immune to things like disreputable/fraudulent researchers, at least we don't have to worry about p-hacking. We still have issues related to the file-drawer phenomena, I'd guess. However, it is not uncommon to come across a study where one of the subjects didn't respond to the initial IV, and the researcher just altered/added something to it (or else went into detail about subject/environment characteristics or interactions that may have affected the results). I can then compare characteristics of my target client and adjust accordingly. It's not perfect, and findings are often misused (e.g., treating them as if they are proof of some universal natural state of human existence, vs. just an inductive-research produced answer to the question "what happened to y when I did x?"), but it seems to have worked out well for us (compare funding for ABA to that of outpatient psych, as well as salaries for masters level credentialled ABA clinicians vs. similarly credentialled MH providers -or even many doctoral level psychologists).

 

[Ollie123]

I don't think it's a badge of shame at all, I just think it should be explicitly stated.

Think we are pretty much on the same page. I 100% agree the things you describe are major problems. I just think we've prematurely jumped on the pre-registration bandwagon and pretend it is the only/optimal solution and are trying to pound the square peg through the round hole to make it work.

What he basically said is because the psyd program doesn't make enough money for the school the faculty are expected to do it all for less money.

This is somewhat amusing. I'm an Assist Prof wearing both research/clinical hats at an AMC in the top 5 for NIH funding, have 2 of my own grants and am Co-I on 2 others, and am getting absolutely slaughtered. I don't think the problem is dollars per se, I think the problem is growing bureaucracy but treating faculty as independent entrepreneurs expected to do things without support, guidance, or even consistency.



In writing this, one thing I would like to see is NIH commit a fixed amount to each grant to offset bureaucracy. Think of it as earmarked indirects. Making it a fixed amount will help keep things in check. One of my main issues facing junior faculty is that we are faced with the same level of bureaucracy that senior faculty have, but with much less support. A lot easier to absorb things on 1 mil/year total directs (2 R01s) than 100k/year directs. I think forcing accountability and ensuring ALL involves parties have some skin in the game and are contributing to solutions would go a very long way towards fixing things.