Death to p values?

[jdawg2017]

I'm curious if anyone saw this letter in Nature yesterday that gathered 800+ scientists to support an approach to ditch statistical significance in favor of what the authors call "compatibility intervals" (basically CIs)? I can understand the idea behind this because science has long been marred by the file-drawer problem that has emerged from a relatively arbitrary distinction that p > .05 is non-significant.

What have more senior folks out there done to address this in your research? In my grad student life I've basically learned that even if p values are not that informative you are still damned in many cases if you don't use them because reviewers will ask for them anyways.

Looking forward to hopefully a good discussion!

---

Members don't see this ad.

 

[WisNeuro]

I'm curious if anyone saw this letter in Nature yesterday that gathered 800+ scientists to support an approach to ditch statistical significance in favor of what the authors call "compatibility intervals" (basically CIs)? I can understand the idea behind this because science has long been marred by the file-drawer problem that has emerged from a relatively arbitrary distinction that p > .05 is non-significant.

What have more senior folks out there done to address this in your research? In my grad student life I've basically learned that even if p values are not that informative you are still damned in many cases if you don't use them because reviewers will ask for them anyways.

Looking forward to hopefully a good discussion!

You still present them, but you just don't make anything of them. We've been in the habit of focusing on appropriate effect sizes and clinically meaningful change over statistical significance for some time now.

 

[Ollie123]

I strongly believe we should all have moved to Bayesian approaches decades ago, which somewhat obviates these concerns.

Have I ever done this in my own work? No. In large part because: A) Reviewers expect frequentist methods; B) Bayesian methods are much less accessible to readers; C) Bayesian methods for complex designs are exponentially more difficult and not well-established; and D) Despite having more training in it than, I would wager, 99% of clinical psychologists - I still really lack the training to be comfortable - especially for more complicated study designs.

Part of the issue with p values is that people want that binary decision point. Statisticians and researchers are comfortable with hedging and dealing in complexity. Policymakers, medical providers, business owners and others hate it and want a binary decision. P values, for better or worse, are a way of "dumbing things down" for the masses.

So basically - I agree. And I don't think this article will work. Because the same damn article has been written literally dozens of times in dozens of different journals. Literally since the 1980's (and possibly earlier).

 

[futureapppsy2]

You still present them, but you just don't make anything of them. We've been in the habit of focusing on appropriate effect sizes and clinically meaningful change over statistical significance for some time now.

You'd be surprised. We published a systematic lit review a couple of years ago where the samples tended to be small n (sub-100, subpopulation of a subpopulation) and our analysis focused on effect sizes (that we calculated ourselves because none of the articles provided them). Reviewer comments requested that we give more weight/focus to the results of the statistical significance testing reported by the original articles.

 

[WisNeuro]

You'd be surprised. We published a systematic lit review a couple of years ago where the samples tended to be small n (sub-100, subpopulation of a subpopulation) and our analysis focused on effect sizes (that we calculated ourselves because none of the articles provided them). Reviewer comments requested that we give more weight/focus to the results of the statistical significance testing reported by the original articles.

Oh, I'm not surprised. I should have clarified the "we've" I my statement. I meant the studies that I publish. I am no longer surprised by the stupid and inane things that come back with reviewer's comments at times.

 

[ClinicalABA]

A quarter century after publication and this is article is still just as relevant:

http://www.iro.umontreal.ca/~dift3913/cours/papers/cohen1994_The_earth_is_round.pdf

(not sure the link works automatically, but if you cut and paste into a browser it should get you there. It's to this article:

Cohen, J. (1994). The earth is round (p<.05). American Psychologist.

While significance testing and other techniques employing inferential statistics certainly have their place, I'm happy to be working in field that almost exclusively avoids them.

 

[futureapppsy2]

A quarter century after publication and this is article is still just as relevant:

http://www.iro.umontreal.ca/~dift3913/cours/papers/cohen1994_The_earth_is_round.pdf

(not sure the link works automatically, but if you cut and paste into a browser it should get you there. It's to this article:

Cohen, J. (1994). The earth is round (p<.05). American Psychologist.

While significance testing and other techniques employing inferential statistics certainly have their place, I'm happy to be working in field that almost exclusively avoids them.

You've never gotten into an argument about parametric effect sizes versus non-parametric effect sizes versus visual analysis in single-case research?

 

[deleted343839]

Cohen, J. (1994). The earth is round (p<.05). American Psychologist.

A classic! This was assigned reading in my first semester of grad school.

 

[Fan_of_Meehl]

I'm curious if anyone saw this letter in Nature yesterday that gathered 800+ scientists to support an approach to ditch statistical significance in favor of what the authors call "compatibility intervals" (basically CIs)? I can understand the idea behind this because science has long been marred by the file-drawer problem that has emerged from a relatively arbitrary distinction that p > .05 is non-significant.

What have more senior folks out there done to address this in your research? In my grad student life I've basically learned that even if p values are not that informative you are still damned in many cases if you don't use them because reviewers will ask for them anyways.

Looking forward to hopefully a good discussion!

Basic science has been marred before. The accounts of its death have been greatly exaggerated.

 

[Fan_of_Meehl]

Jacob Cohen is a badass.

 

[Fan_of_Meehl]

I'm curious if anyone saw this letter in Nature yesterday that gathered 800+ scientists to support an approach to ditch statistical significance in favor of what the authors call "compatibility intervals" (basically CIs)? I can understand the idea behind this because science has long been marred by the file-drawer problem that has emerged from a relatively arbitrary distinction that p > .05 is non-significant.

What have more senior folks out there done to address this in your research? In my grad student life I've basically learned that even if p values are not that informative you are still damned in many cases if you don't use them because reviewers will ask for them anyways.

Looking forward to hopefully a good discussion!

Jacob Cohen is an eternal badass.