Patients at Risk: The Rise of the Nurse Practitioner and Physician Assistant in Healthcare

This forum made possible through the generous support of SDN members, donors, and sponsors. Thank you.
That is still conflating an argument about:

1) the prevailing statistical methods used in null hypothesis testing
WITH
2) the general concept of the null hypothesis.

Let me try to put forth an analogy. To my understanding, the initial vitamin D literature was skewed because of some mathematical errors. Then it was corrected. That's a statistical error. If I said that vitamin D is basically a non-issue, and cited that error, you'd probably express frustration that I am conflating the importance of a concept with an apt statistical methodological problem. That's what I'm trying to point out.

Right. That paper that I posted actually takes issue with number 2 much more directly than number 1, although it engages with the later briefly.


Okay, follow up:

1) Define which theoretical setting the research question falls within. Is this set theory, model theory, etc? This changes everything. Closed set? Then Friedman would agree with you, but you'd have a lot of other methodological problems.
I genuinely don't understand this question, sorry, I'll have to ask for clarification as to how it connects to what I said.



2) Is the actual argument that the CONCEPT of the null hypothesis is wrong? If so, then why was Gelman cited?

The concept is not wrong in the formal mathematical sense within frequentist statistics; rather, given the structure of the world and the nature of the phenomena and measurements generally under study in biomedical and social science fields, it is an irrelevant, misleading, and not useful concept.

Genetics might be an exception, among other things there might plausibly be genuinely zero impact of a given particular SNP on a particular protein's expression or some other process of interest.

3) Is the actual argument that the prevailing statistical methods in null hypothesis significance testing is wrong? If so, then reconcile the stated opinion about the work cited with Gelman's statements that the prevailing methods should be included, but not retain the same importance.

I think you want to look at that paper again. They state in several sections in several different ways that much of the time statistical analysis probably shouldn't even result in p-values. It is not a proposal to slightly tweak existing methods. They also say that using any statistical threshold to decide whether something is not is not significant in a dichotomous way has the same problems.

It's not null hypothesis significance testing if you are not making a yes/no determination of significance. If you are comparing effect sizes, you do not need NHST. If you are assessing fit with a multi-level model, you do not need NHST. If you are postulating a causal graph a la Judah Pearl, you do not need NHST.

4) Or is this all just a debate style? I'm fine if it is.

No, this is substantive.
 
I'll try to keep this in order, but there are challenges. And I'm a dilletante. @AbnormalPsych or dozens of other professors could correct me or explain it better.

My read of Gelman is that he believes that publishers should not require someone has to show statistical significance as a prerequisite, before allowing authors to move to more advanced statistical analysis (i.e. why require statistical significance before allowing other metrics, why not just allow all metrics to be used). This stance is consistent with the ASA publications about this matter. This argument is about the error in how the concept of the null hypothesis is administered. It is not an argument stating that the whole concept is erroneous. Which you disagree with in your initial point, and then contradict yourself in your latter points.

That leaves out Gelman's points about the error of dichotomous decisions vs. decisions along a continuum. This is also a non-issue. Science has established how to statistically handle this matter for more than a century. Any of the physiology literature has continuous decisions, using pretty fun methods. So does economic literature. So does fluid dynamics. etc.

In regards to his proposition to use Bayes most powerful: I get it. You have advanced education in respiratory physiology. Of course you love anything written by Laplace. Gelman assumes that problems have many potential answers, and a priori assumptions are inevitable, which makes Bayes most powerful a given. However, there is considerable debate as to whether Bayes most power represents an inductive model of reasoning or deductive model of reasoning. Because setting determines appropriateness, this is exceptionally important. For example, if I showed up to a neurological examination with a sledge hammer, you might point out that this is the wrong type of hammer. Mathematical logic has several different divisions. Broad strokes, there are two or more gross options as I see it:

1) Our professions are based upon theories (e.g., materialism, every variable is known or knowable). Then we are operating on a priori assumptions, which requires deductive reasoning (i.e., starting at some general theory/hypothesis, and determining if the data "fit" with these hypotheses). This would be model theory and the opposite of how Bayes most powerful operates. So Bayes would be inappropriate for this form of deductive reasoning.
2) Our professions are limiting themselves to generally agreed upon sets of data or study, the a priori assumptions offered by Bayes are somehow exempt as evidence of a hypothesis and deductive reasoning, and I still don't understand how set theory is not a branch of model theory. That being the case, we are blind men attempting to figure out the elephant through induction. If this is the case, then the inductive model used by Bayes is appropriate, and then null hypothesis is useless. However, if this is true then Bayes cannot prove or support anything. It can only describe. Which would then require the user to go back, create some theory or model by which to make the required a priori assumptions, which would subsequently be again subject to testing of the null hypothesis and other methods of deductive reasoning.

Or maybe I have no idea what I'm talking about. I dunno.
 
I'll be honest. I got pretty lost and confused while reading this thread earlier in the week and the arguments being made and evidence to support those meandering arguments, and have just tuned out at this point.

Part of this might be the libations flowing, so then my bad. :shrug:

I agree with what PsyDr seems to be saying above regarding deductive, a priori assumptions, etc etc.

Then again, I'm not a physician, so I have no idea what kind of, or, the number of years of advanced research methodology and design and statistical analysis training you all get / have. Maybe you know more. Maybe not.
 
I'll be honest. I got pretty lost and confused while reading this thread earlier in the week and the arguments being made and evidence to support those meandering arguments, and have just tuned out at this point.

Part of this might be the libations flowing, so then my bad. :shrug:

I agree with what PsyDr seems to be saying above regarding deductive, a priori assumptions, etc etc.

Then again, I'm not a physician, so I have no idea what kind of, or, the number of years of advanced research methodology and design and statistical analysis training you all get / have. Maybe you know more. Maybe not.

I did a cog neuro PhD prior to med school so I suspect my formal grad school stats/research methodology training was fairly similar to yours. The points I was making are very basic and conceptual and I still haven't read a super germane response, so I am with you as far as following meandering arguments are concerned. I struggle to understand especially how anyone reading the linked paper can miss how it repeatedly and in several ways points out the basic weaknesses of NHST but @PsyDr 's responses make it clear it is not signposted enough as apparently he doesn't see any of the relevant passages.

Regardless, when this starts being less a discussion about ideas and more sniping about different professions and their training standards in lieu of engaging with ideas, time for me to bow out.
 
I think you want to look at that paper again. They state in several sections in several different ways that much of the time statistical analysis probably shouldn't even result in p-values. It is not a proposal to slightly tweak existing methods. They also say that using any statistical threshold to decide whether something is not is not significant in a dichotomous way has the same problems.

@PsyDr is right. Selecting the appropriate prior has a large effect on the results and there is not a method available for appropriate prior selection. This is why few people I know of are advocating for such a purist approach. It's much more common to advocate for integrative approaches (see article below).

 
@PsyDr is right. Selecting the appropriate prior has a large effect on the results and there is not a method available for appropriate prior selection. This is why few people I know of are advocating for such a purist approach. It's much more common to advocate for integrative approaches (see article below).


Apparently my impulse control wrt responding to arcane discussions is poor but this a thoughtful and clear response so I couldn't help attempting a clear and thoughtful answer. You're absolutely right that there's no mechanistic way to set priors in a way that is neutral and impartial. It is absolutely putting a thumb on the scales in a sense. I think there is a deep philosophical difference between people who hear that and are horrified and people like me who sort of shrug and accept that it is not really possible to usefully evaluate any particular result without consideration of the totality of evidence. I accept that means different investigators will reach different conclusions and there will not be a simple way to decide between them.

I don't think there's a way around reasoning and argumentation, flawed and biased in many ways as they may be. Our colleagues in harder sciences avoid this by just requiring extremely high standards of evidence, e.g. in particle physics measurements have to differ by five SDs to be considered truly different. They barely need inferential statistics.

I don't see a way around embracing an amount of uncertainty and epistemic humility in the degree of confidence we can have in our conclusions. I think evidence from other results and sources will significantly impact how much weight we assign to a given new result and this may reasonably differ from scholar to scholar. That is closer to my conception of the basic task of inferential reasoning. I am aware anf can accept others have a different conception of the structure of the problem.
 
I did a cog neuro PhD prior to med school
I knew it. One of us! One of us! 🤓

clausewitz2 said:
so I suspect my formal grad school stats/research methodology training was fairly similar to yours.

Not to butt in but my impression is that there is something unique about the culture of statistical training in psychology. They seem to have certain methods and approaches that are used only in that discipline. Memorably, last year I was involved in a project in which there was a huge kerfuffle between the psychologists and the statistician about the approach that I frankly barely understood, but which resulted in the statistician quitting the project in a huff 🙁
 
I knew it. One of us! One of us! 🤓



Not to butt in but my impression is that there is something unique about the culture of statistical training in psychology. They seem to have certain methods and approaches that are used only in that discipline. Memorably, last year I was involved in a project in which there was a huge kerfuffle between the psychologists and the statistician about the approach that I frankly barely understood, but which resulted in the statistician quitting the project in a huff 🙁

I wouldn't caution against the representativeness heuristic on this one, but I will say that we're trained to approach things within the bounds of theory via logical (post) positivism. I've encountered a few psychologists who use theory as a stick to force round data into square pegs. I'm sure quite a few psychologists find this equally detestable as any statistician would.
 
I wouldn't caution against the representativeness heuristic on this one, but I will say that we're trained to approach things within the bounds of theory via logical (post) positivism. I've encountered a few psychologists who use theory as a stick to force round data into square pegs. I'm sure quite a few psychologists find this equally detestable as any statistician would.
Can you say more about this? When you say logical positivism, I think Wittgenstein/Vienna Circle, which seems singularly, in fact intentionally, useless for making any sort of practical scientific inference whatsoever. This can't be what you mean though?
 
Can you say more about this? When you say logical positivism, I think Wittgenstein/Vienna Circle, which seems singularly, in fact intentionally, useless for making any sort of practical scientific inference whatsoever. This can't be what you mean though?

Others can correct me, but my understanding is that psychologists usually fall into either humanism or positivism in their approaches to research. Post positivism is the amendment to logical positivism that makes psychological research more tenable. In the (post) positivist camp, theoretical assumptions are treated as a priori to inform a research question (which is one for the reasons the intro sections in our journals are so d*** long) and the question being answered is answered within that theoretical paradigm. I included some references below to capture the current epistemological debates in psychology. Some of these are from my history of psychology seminar in graduate school and others I found online.





 
Last edited:
Regardless, when this starts being less a discussion about ideas and more sniping about different professions and their training standards in lieu of engaging with ideas, time for me to bow out.

It was not meant to be sniping, so apologies, it is just pointing out there are real differences in the approach and methods used by professions and the training/tools/philosophy getting each of us there.

See:

Not to butt in but my impression is that there is something unique about the culture of statistical training in psychology. They seem to have certain methods and approaches that are used only in that discipline. Memorably, last year I was involved in a project in which there was a huge kerfuffle between the psychologists and the statistician about the approach that I frankly barely understood, but which resulted in the statistician quitting the project in a huff

Also, see @R. Matey 's posts above.

I wouldn't caution against the representativeness heuristic on this one, but I will say that we're trained to approach things within the bounds of theory via logical (post) positivism. I've encountered a few psychologists who use theory as a stick to force round data into square pegs. I'm sure quite a few psychologists find this equally detestable as any statistician would.

Others can correct me, but my understanding is that psychologists usually fall into either humanism or positivism in their approaches to research. Post positivism is the amendment to logical positivism that makes psychological research more tenable. In the (post) positivist camp, theoretical assumptions are treated as a priori to inform a research question, (which is one for the reasons the intro sections in our journals are so d*** long) and the question being answered is answered within that theoretical paradigm.

Some references:

https://www.sas.upenn.edu/~hatfield/hphxppsych.pdf
https://www.tandfonline.com/doi/abs/10.1080/0951508021000042030
 
Last edited:
Top