PhD/PsyD Dissertation stuck due to power analysis

[psydstudent2020]

I’m not sure if this is allowed on here but I hope so because I’ve received solid advice on this site before. My dissertation isn’t moving forward until I do a power analysis. However, my chair does not know how to do it for the statistical tests I am running. The other member of my committee said I don’t need one, and my third is the one who said I NEED to do it but he is off campus and when I emailed about it he asked me questions like he didn’t remember my study even though he wrote most of the proposal revisions. I asked psychologists/statisticians outside my committee but they didn’t know either. My study is on the development of a self report inventory. It has 7 items but may have more or less because step one after submitting to the IRB is to send it to psychologists for review based on their clinical experience. The statistical tests that will be run: 1) split half reliability, 2) internal consistency using cronbachs alpha, 3) test retest reliability, 4) differential validity (giving a well known personality questionnaire along with my instrument). Would there need to be one power analysis done for all this, or one for each test? Also, how do I actually do the power analysis? I know what G power is but that is where everyone gets stuck because they say you can’t do my specific statistical tests on there. Please message me if you can help. Thank you and I’m sorry if this isn’t allowed!

---

Members don't see this ad.

 

[WisNeuro]

This would constitute homework help, however, some people may be able to help you by suggesting readings about power analyses.

 

[PsyDr]

I’ll suggest hiring one of those statisticians in the back of the apa trade publication.

 

[psydstudent2020]

I’ll suggest hiring one of those statisticians in the back of the apa trade publication.

Could you PM me their contact info? I have no idea who they are or how to look them up. Thank you!

 

[PsyDr]

I have no idea who they are. They advertise in the back of the apa monitor or online on the apa site.

 

[Ollie123]

Not sure I'd consider this homework help per se. Seems reasonable to consult on some things and sounds like they have already made some effort to address themselves.

You typically do a power analysis for each test (or at least should). The problem is that you are describing methodological approaches and not statistical techniques. i.e. "Differential validity" isn't a statistical test. It depends a little on what you are doing, but 1 and 3 might just be a pearson r under conventional approaches, which is in g power. 4 it depends on what you are doing. You won't do a power analysis for cronbach's alpha because its descriptive and not inferential.

If you reframe these as actual statistical tests, anyone in your local stats/biostats department should be able to help you with the rest. As power analysis goes, what you want to do is likely to be about as simple/straightforward as it can get.

 

[WisNeuro]

If it's simply a power analysis, you can pretty much figure this out reading a few articles/chapters. Pretty sure you can find COhen's second edition online in full. Otherwise Paul Ellis has a decent one, and there are hundreds of articles that will help understand this. Quick search on google scholar points to dozens upon dozens of free articles that go into this. If it's for your dissertation, I'd suggest figuring this out so that you can actually answer questions if your committee asks. If you have someone else do it for you, you'll likely have a hard time explaining what was done, or why it was done.

 

[FreudianSlippers]

It sucks that you are not getting more support through your program. However, I agree with @WisNeuro and the tests you mentioned shouldn't be hard at all. What got me through a lot of tricky stats and data analysis problems was actually posting my questions on data analysis software forums (e.g. STATA, R, SPSS). In my experience, people there are super nerdy and friendly and are more than willing to put you in the right direction.

Also, you will definitely need to do one at a time. Just google "how to do [split half reliability] using [data analysis software]". There is an art to data analysis, and there's usually many answers to one question. What's the best way to eat an elephant? One bight at a time.

 

[psydstudent2020]

Not sure I'd consider this homework help per se. Seems reasonable to consult on some things and sounds like they have already made some effort to address themselves.

You typically do a power analysis for each test (or at least should). The problem is that you are describing methodological approaches and not statistical techniques. i.e. "Differential validity" isn't a statistical test. It depends a little on what you are doing, but 1 and 3 might just be a pearson r under conventional approaches, which is in g power. 4 it depends on what you are doing. You won't do a power analysis for cronbach's alpha because its descriptive and not inferential.

If you reframe these as actual statistical tests, anyone in your local stats/biostats department should be able to help you with the rest. As power analysis goes, what you want to do is likely to be about as simple/straightforward as it can get.

Thank you! I don’t know what tests we will be running. I did run GPower for correlation as you suggested. In the revisions, the committee member stated that he wanted a power analysis for cronbachs alpha, which is partly why I’m confused. How would you determine sample size for the alpha then?

 

[WisNeuro]

Also, do they not teach power analysis in grad stats anymore?>

 

[psydstudent2020]

Also, do they not teach power analysis in grad stats anymore?>

They didn’t teach it in my class. I taught myself most of how to work GPower but I’ve been told that can’t do all of the analyses I need.

 

[ClinicalABA]

I'm not trying to be snarky here- honestly- but did you do a google search of "power analysis for Cronbach's alpha"? I just did. Looks like there are some very relevant articles, so I bet doing and actual literature search in a psychinfo type database will put you on the right path. (Seriously- I typed "power analysis for C" and it autofilled "onbach's alpha" for me!). I was even able to download full text of some of the articles and a quick perusal of one of them seems to indicate that you can derive power functions and formulas from Cronbach's alpha, with greater Ca's related to greater power. And- the article is talking about questionnaire data! Again- I'm really not trying to be snarky, but I found that info in about 5 minutes just using google!


Now, that said, you will need to have some decent understanding of some statistical concepts to get through some of the articles. However, this is your dissertation- It's meant to serve as proof of your aptitude with the scientific method. I'd suggest you look at some of those articles, slog through the stat-talk, and then set up a time to talk with a statistics professor at your school for clarification. You may, unfortunately, discover that any stat programs you find do not not have the exact functions you need built in. In that case you might need to do some transformations of what is available. If all that fails- and your program allows it- hire a stats person to do what you need done.

 

[psydstudent2020]

I'm not trying to be snarky here- honestly- but did you do a google search of "power analysis for Cronbach's alpha"? I just did. Looks like there are some very relevant articles, so I bet doing and actual literature search in a psychinfo type database will put you on the right path. (Seriously- I typed "power analysis for C" and it autofilled "onbach's alpha" for me!). I was even able to download full text of some of the articles and a quick perusal of one of them seems to indicate that you can derive power functions and formulas from Cronbach's alpha, with greater Ca's related to greater power. And- the article is talking about questionnaire data! Again- I'm really not trying to be snarky, but I found that info in about 5 minutes just using google!


Now, that said, you will need to have some decent understanding of some statistical concepts to get through some of the articles. However, this is your dissertation- It's meant to serve as proof of your aptitude with the scientific method. I'd suggest you look at some of those articles, slog through the stat-talk, and then set up a time to talk with a statistics professor at your school for clarification. You may, unfortunately, discover that any stat programs you find do not not have the exact functions you need built in. In that case you might need to do some transformations of what is available. If all that fails- and your program allows it- hire a stats person to do what you need done.

I have googled the info and brought it back to my committee, but they want it done through a specific program. I reached out to them again today so hopefully I get more assistance.

 

[WisNeuro]

The specific program for power analyses is irrelevant, as long as the underlying power analysis is the appropriate one for the task. Heck, a lot of these can be done by hand in not much time.

 

[psydstudent2020]

Could someone direct me to an article for each of the specific things I numbered? I’m pretty sure it will be a correlation for all of them from everything I have read about each of them and the SPSS videos I’ve watched in prep for data analysis. I would love to learn how to do it! Problem is I can’t find anything specific to my study so I get referred to ask someone else.

 

[psydstudent2020]

The specific program for power analyses is irrelevant, as long as the underlying power analysis is the appropriate one for the task. Heck, a lot of these can be done by hand in not much time.

I wouldn’t mind doing them by hand. I also realize it shouldn’t take this long to even get the power analysis done. Are there formulas or articles you recommend?

 

[ClinicalABA]

I have googled the info and brought it back to my committee, but they want it done through a specific program. I reached out to them again today so hopefully I get more assistance.

In that case, I hope they can give you more specific guidance on how to do it with that specific program. Best of luck with the dissertation!

Question for some of the more stat-savvy folks out there- Can you determine specifically how you'll measure chance of type II error (power) without knowing how you specifically measure chance of type I error (statistical significance)? This all is little curious to me in that the OP states s/he has been asked to calculate power but has not yet determined what types of significance testing s/he will be doing. It's been awhile since I've dabbled in such things, and programs differ in there dissertation requirements and processes, but I would think identifying the statistics you are going to use related to significance typically is of greater importance to power analyses, particularly in the case of 7 planned comparisons.

 

[WisNeuro]

I wouldn’t mind doing them by hand. I also realize it shouldn’t take this long to even get the power analysis done. Are there formulas or articles you recommend?

http://www.utstat.toronto.edu/~brunner/oldclass/378f16/readings/CohenPower.pdf

 

[ClinicalABA]

Could someone direct me to an article for each of the specific things I numbered? I’m pretty sure it will be a correlation for all of them from everything I have read about each of them and the SPSS videos I’ve watched in prep for data analysis. I would love to learn how to do it! Problem is I can’t find anything specific to my study so I get referred to ask someone else.

I'd start by doing a lit search for any articles related to scale or questionnaire construction and see what stats they used to measure the variables you are looking at. I'd imagine any such articles would've looked at the issues of reliability and validity that you are concerned with.

 

[psydstudent2020]

http://www.utstat.toronto.edu/~brunner/oldclass/378f16/readings/CohenPower.pdf

Thank you!

 

[psydstudent2020]

Thanks! So far it looks like correlations for all.

 

[psydstudent2020]

Went back to notes from my proposal - correlations were mentioned for each of these except for cronbachs alpha which was item analysis

 

[ClinicalABA]

Thanks! So far it looks like correlations for all.

That should make power calculations pretty straightforward then. You can probably even find on-line calculators where all you need to do is enter alpha and sample size.

 

[Ollie123]

We learned power analysis in grad school, but unfortunately its usually taught in an overly simplistic way that almost never covers the type of work I actually do. Statisticians are even worse and usually confused by anything that isn't a clinical trial. I'm convinced about 95% of power analyses are done incorrectly and Gpower can do a lot less than most people think. Given the number of assumptions that get baked into power analyses, I'm fairly well convinced its just a mindless ritual done to appease grant reviewers. I had a dream of building a full set of simulation tools that let you do an <actual> power analysis for any type of analysis you might attempt in R, but realistically I doubt I'll ever get to it.

If they are correlations than you do the power analysis for correlations. Simple as that. This is doable in Gpower.

Perhaps I'm nit-picking, but it shouldn't technically be "power analysis" for Cronbach's alpha. There are ways to determine an appropriate sample size, but without an inferential test I'm just not sure what you would be "powering." Same deal with factor analysis. There are rough guidelines, its a field unto itself and quite complicated. Most people just follow conventions.

This conversation does worry me a bit. No one should ever say

I don’t know what tests we will be running

about their own dissertation.

 

[deleted343839]

I had a dream of building a full set of simulation tools that let you do an <actual> power analysis for any type of analysis you might attempt in R, but realistically I doubt I'll ever get to it.

Bet you could crowdfund that. I'm in.

SPSS videos I’ve watched in prep for data analysis

I'm curious what kind of stats training or coursework you've had. Sorry you're not getting more support for this. I'm leaning in favor of others' advice to hire a stats consultant if possible. The good ones will teach you and walk you through what they did and why.

 

[EmotRegulation]

In addition to an analysis in GPower, there are articles to cite about sufficient sample size for measure development, as no one has seemed to mention that here yet. Usually the idea is that you need at least 10 people per item, so the "minimum" sample size will vary based on the initial number of items in the scale, though not everyone agrees on that. More people is usually better. Here are a few papers, but please do your own work and look at the variety of scale development resources out there for info on determining sample size. AND run some analyses on power for your data, per suggestions above (and yes, these are pretty much all correlations....most of measurement development is correlational types of analyses along with factor analysis).

Best Practices for Developing and Validating Scales for Health, Social, and Behavioral Research: A Primer

Scale development and validation are critical to much of the work in the health, social, and behavioral sciences. However, the constellation of techniques required for scale development and evaluation can be onerous, jargon-filled, unfamiliar, and resource-intensive. ...

www.ncbi.nlm.nih.gov

Sample size used to validate a scale: a review of publications on newly-developed patient reported outcomes measures

New patient reported outcome (PRO) measures are regularly developed to assess various aspects of the patients’ perspective on their disease and treatment. For these instruments to be useful in clinical research, they must undergo a proper psychometric ...

www.ncbi.nlm.nih.gov