Psychological Science Accelerator

[WisNeuro]

An interesting concept that combines a few things that we should already be doing as a field. One, pre-registration. Gets around people just p-hacking junk publications. Two, reproducibility. different labs, different conditions, at least some of the PIs won't necessarily be emotionally invested in the outcome they want to happen, presumably. Thoughts?

Can Teamwork Solve One Of Psychology’s Biggest Problems?

---

Members don't see this ad.

 

[smalltownpsych]

It is also a good thing that this would be cross-cultural. I am always curious about what is more universal about human psychology verses what is more culturally specific. I have always enjoyed seeing studies from other countries when searching the literature because of this. Another thought is that we might start learning more about effects of socio-cultural perspectives on interventions for mental health that might predict better outcomes than our society’s focus on pharmacology.

 

[PsyDr]

IMO: psychology is currently lacking in interesting and broad enough questions, not data. But maybe having data and working backwards is a better approach.

 

[Fan_of_Meehl]

An interesting concept that combines a few things that we should already be doing as a field. One, pre-registration. Gets around people just p-hacking junk publications. Two, reproducibility. different labs, different conditions, at least some of the PIs won't necessarily be emotionally invested in the outcome they want to happen, presumably. Thoughts?

Can Teamwork Solve One Of Psychology’s Biggest Problems?

"Hypotheses are always tested in bundles" - Philip Kitcher

Unfortunately, many people forget that their operationalization of a certain key construct (e.g., a raw score on a self-report measure of symptoms such as the PCL-5 (Posttraumatic Stress Disorder Checklist for DSM-5)) can be problematic in that it is not only measuring the construct of interest (psychological symptoms reasonably attributable to PTSD) but also measures other things such as response style (over or under-reporting), symptoms (e.g., insomnia) attributable to causes other than PTSD, degree of interest in increasing service-connection percentage and disability payments, etc.

Even in the natural sciences, hypotheses are always tested 'in bundles' in that the empirical observations themselves rely on certain fundamental assumptions (e.g., laws of optics, thermodynamics, etc.) that are often unacknowledged. This is much more problematic in the social sciences where many of our underlying assumptions have been shown to be faulty but, nonetheless, are often conveniently ignored when it comes time to interpret a string of studies and draw general conclusions.

The link is a great resource and reminder.

 

[Ollie123]

Interesting notion, could help in certain subareas. Largely those where studies can be done very inexpensively (social psych vignette studies, perception/cognition studies using behavioral outcomes). I'm not sure how viable it is for clinical research where we are more reliant on grant support and there is no clear mechanism for how this would be funded.

I'm actually not a fan of the direction all this is heading, despite my being (in my opinion) a pretty vocal complainant about the lack of rigor in science. I think all of what folks are proposing (pre-registration, etc.) is the proverbial band-aid on the broken limb. It is dramatically increasing regulatory burden on investigators since neither NIH nor institutions wants to cough up the resources to really assist with it other than hiring a bunch of people to yell at me for not doing things that no one seems to actually know how I should do or even if I should actually do them. I don't take any of the stuff (IRB, scientific review panels, clinical trials.gov, etc.) seriously anymore, it just seems like bureaucrats yammering and increasing the background noise in my actual day.

The system is broken. If we want to fix it, we need to stop the publish/perish mentality. We need to refocus on quality and discovery rather than quantity and grant dollars. We're putting the cart before the horse and its just increasing pressure on investigators to comply with various demands when the actual incentive structure around keeping their jobs and building their career are still moving in the opposite direction.

 

[WisNeuro]

I agree with many of the sentiments. Although, I hardly see any amelioration of the publish or perish mindset any time in the near future. Federal research grant money has been inversely tracking anti-science sentiment in this country, with no signs of slowing. And, private money won't fund anything unless it's big, and will make money in a relatively short time frame. The system is broken, but it won't be fixed any time soon. So, we can work within a broken system, or wait to see how long it takes to burn itself down.

 

[Ollie123]

I agree with many of the sentiments. Although, I hardly see any amelioration of the publish or perish mindset any time in the near future. Federal research grant money has been inversely tracking anti-science sentiment in this country, with no signs of slowing. And, private money won't fund anything unless it's big, and will make money in a relatively short time frame. The system is broken, but it won't be fixed any time soon. So, we can work within a broken system, or wait to see how long it takes to burn itself down.

Fully agreed. I just question whether things like this are more helpful or harmful in the long run. I don't know the answer to that question (or if there is one). I do know that most of my junior-faculty colleagues in medical centers are frustrated, feeling pulled in multiple directions at once by these matters and feeling like they don't have the resources that senior folks do to implement many of the new 'scientific rigor' initiatives or face the increasing regulatory burden. Several have already said screw it and gone to big pharma or other industry gigs. Others talk about it on a daily basis. I'm one of the only junior folks in my division still hellbent on making an academic career work. We need to make sure we don't drive all the early career folks out of the field (or severely hamper their career progression) if the goal is to produce the best science.

 

[WisNeuro]

. We need to make sure we don't drive all the early career folks out of the field (or severely hamper their career progression) if the goal is to produce the best science.

I fear we've already been doing that for some time. I once thought of an academic career. Starting faculty positions nowadays would run me about a 40% reduction in pay, at best, with much longer hours. There is just little incentive for many to go that route when they can make far more and work far less in other positions.

 

[DynamicDidactic]

The system is broken. If we want to fix it, we need to stop the publish/perish mentality. We need to refocus on quality and discovery rather than quantity and grant dollars. We're putting the cart before the horse and its just increasing pressure on investigators to comply with various demands when the actual incentive structure around keeping their jobs and building their career are still moving in the opposite direction.

I agree with many of the sentiments. Although, I hardly see any amelioration of the publish or perish mindset any time in the near future. Federal research grant money has been inversely tracking anti-science sentiment in this country, with no signs of slowing. And, private money won't fund anything unless it's big, and will make money in a relatively short time frame. The system is broken, but it won't be fixed any time soon. So, we can work within a broken system, or wait to see how long it takes to burn itself down.

I agree that there has been a large increase in perverse incentives for research in recent decades. However, the way academia is structured makes it rather easy to right the ship. Search committees and tenure committees have all the power at each institution to reward quality of research rather than quantity of pubs/funds.

 

[WisNeuro]

I agree that there has been a large increase in perverse incentives for research in recent decades. However, the way academia is structured makes it rather easy to right the ship. Search committees and tenure committees have all the power at each institution to reward quality of research rather than quantity of pubs/funds.

Search committees are still beholden to higher powers, and I assume those higher powers care about grant monies coming in a great deal.

 

[deleted343839]

I agree that there has been a large increase in perverse incentives for research in recent decades. However, the way academia is structured makes it rather easy to right the ship. Search committees and tenure committees have all the power at each institution to reward quality of research rather than quantity of pubs/funds.

I believe this is less true today than at any other point in the modern history of academia.

 

[DynamicDidactic]

I believe this is less true today than at any other point in the modern history of academia.

Search committees are still beholden to higher powers, and I assume those higher powers care about grant monies coming in a great deal.

Can you please elaborate? My experience, from both ends of the search, it seems the departments still hold much of the power (in traditional university settings).

 

[WisNeuro]

Can you please elaborate? My experience, from both ends of the search, it seems the departments still hold much of the power (in traditional university settings).

It's been a few years since I've had this conversation with a colleague (public state college), but they were pretty clear that in searching for faculty hires, potential to draw grants was one of the biggest factors in a hire due to pressure from the upper level admin to be bringing in a certain level of extramural funding, that seemed to be ever increasing. I don't know if that is still the case, but, considering the state that they are in, and it's recent state level cuts to higher education, I can't imagine that that trend has reversed.

 

[DynamicDidactic]

It's been a few years since I've had this conversation with a colleague (public state college), but they were pretty clear that in searching for faculty hires, potential to draw grants was one of the biggest factors in a hire due to pressure from the upper level admin to be bringing in a certain level of extramural funding, that seemed to be ever increasing. I don't know if that is still the case, but, considering the state that they are in, and it's recent state level cuts to higher education, I can't imagine that that trend has reversed.

I agree that there is pressure, but the department still chooses the candidate for approval by administration. Meaning the department continues to choose the type of scholarly work they will reinforce.

 

[Ollie123]

Worth clarifying a couple things:
1) Academia comes in many forms. Many in medical schools are 100% soft money. Joint appointments between psychology and medicine are becoming increasingly common and often offer a percentage hard money (e.g. 50%) with the remainder soft. Independent institutions (e.g. research institutes, freestanding cancer centers) also often operate that way. These represent a sizable portion of academic psychologists and a very sizable portion of overall psychology research output. A search committee can say "grants don't matter" all they want, but it means nothing. Folks will be gone or no longer conducting research inside 2-3 years.
2) Hires (and especially promotion/tenure) typically requires sign-off from admin at multiple levels (e.g. department, college, university). At least that is how its worked at every institution I've been a part of, even in traditional psychology departments. They don't often overrule the department....but departments don't often put forward people without funding or substantive publication records.

There isn't going to be an easy solution here. Everything is intertwined (rising costs of college education, increased competition for federal funding, administrative "bloat" and regulatory requirements). This even crosses departments since there is often shared infrastructure between them. Make a change that impacts medical schools? Any psych department faculty doing fMRI is likely to be affected since very few psych departments have their own scanners. Same goes for clinical trials (auditing, compliance, data security, etc. is often housed within medicine...even if its a behavioral study). I agree the system is broken. I think a lot of the directions we are pushing are extremely naive and short-sighted solutions to much bigger issues. I think many are likely to make the problem worse rather than better. I think what we need is some change agents/early adopters in government and academia, but no one seems willing to take the risk on any meaningful reform (this is true about far more than just academia). Instead its just "Let's have investigators fill out 50 additional redundant forms wherein they pinky swear they will do the best science and totally won't have their secretaries fill out the forms for them, then set up a pay structure so their salaries are directly linked to the size of the p-values they obtain."

I could go on about this for a while I'm trying to get funding to directly examine/tackle some of these system issues. We'll see what happens.

 

[WisNeuro]

I agree that there is pressure, but the department still chooses the candidate for approval by administration. Meaning the department continues to choose the type of scholarly work they will reinforce.

True, but I imagine there is a ton of pressure, that will undoubtedly influence the hiring decision, to go with a candidate that is doing work that is more "fundable" rather than perhaps an area that the department would like to foster of which there may be fewer funding opportunities.

As Ollie hints to, money is driving these decisions. And, as that pool of available money dwindles, that pressure only intensifies. There is little pressure to reward quality if it doesn't come on the back of a pile of grant money.

 

[DynamicDidactic]

True, but I imagine there is a ton of pressure, that will undoubtedly influence the hiring decision, to go with a candidate that is doing work that is more "fundable" rather than perhaps an area that the department would like to foster of which there may be fewer funding opportunities.

As Ollie hints to, money is driving these decisions. And, as that pool of available money dwindles, that pressure only intensifies. There is little pressure to reward quality if it doesn't come on the back of a pile of grant money.

I agree about the money. And I agree about the pressure. I do want to highlight the role colleagues play in reinforcing the system and the role they can play in changing the system.

 

[deleted343839]

I recommend this book. The general thesis will not come as a surprise to many but the history and broader picture of science funding are interesting (and disheartening): How Economics Shapes Science — Paula Stephan | Harvard University Press