Comparing Dependent Effect Sizes

[Marissa4usa]

---

Members don't see this ad.

I'm trying to find some information on comparing the effect sizes for dependent variables (i.e. I want to know whether the effect size for the relationship between variables X and Y is different from the effect size for the relationship between variables X and Z.

I expected there to be a ton of information on Google but every time I think I got something helpful, it's not what I need. The closest to what I'm looking for was some info on the comparing the effect sizes for independent samples. Maybe I'm not using the right search terms...

If somebody can point me to some articles (or website that will automatically calculate the difference for me 😀 ), I would really appreciate it.

 

[coldsweat]

I'm trying to find some information on comparing the effect sizes for dependent variables (i.e. I want to know whether the effect size for the relationship between variables X and Y is different from the effect size for the relationship between variables X and Z.

I expected there to be a ton of information on Google but every time I think I got something helpful, it's not what I need. The closest to what I'm looking for was some info on the comparing the effect sizes for independent samples. Maybe I'm not using the right search terms...

If somebody can point me to some articles (or website that will automatically calculate the difference for me 😀 ), I would really appreciate it.

What type of dependent effect sizes are you attempting to compare? I'm guessing a correlation coefficient based on your use of the word "relationship," and correlation coefficients are a type of effect size, but I want to verify. If you are comparing dependent correlation coefficients, the resource below will help you:

Zou, G. Y. (2007). Toward using confidence intervals to compare correlations. Psychological Methods, 12(4), 399–413. doi:10.1037/1082-989X.12.4.399

With the method proposed in this article, not only can you determine whether the difference between two dependent correlation coefficients is significant, but you can also calculate a confidence interval for the difference.

I implemented the method proposed in this paper with the cocor package in R. I don't know if other stats packages have canned procedures for performing this analysis.

If I misinterpreted your question please let me know!

 

[PhDToBe]

I'm not good at stats, but this might be a situation where you would use steiger's t-tests.

 

[Marissa4usa]

What type of dependent effect sizes are you attempting to compare? I'm guessing a correlation coefficient based on your use of the word "relationship," and correlation coefficients are a type of effect size, but I want to verify. If you are comparing dependent correlation coefficients, the resource below will help you:

Zou, G. Y. (2007). Toward using confidence intervals to compare correlations. Psychological Methods, 12(4), 399–413. doi:10.1037/1082-989X.12.4.399

I've actually calculated the effect size correlations and run the appropriate test, but I'm specifically looking to compare Cohen's d effect sizes.

It looks like I'll need to explain a bit more since my wording was a bit confusing: I ran two hierarchical linear models for X predicting Y and then X predicting Z. Each yielded a regression coefficient as well as a t-score. From the t-scores I calculated the effect sizes (Cohen's d). I want to know whether these two effect sizes differ from each other (or if there is maybe such thing as an effect size for the difference between the effect sizes).

 

[Ollie123]

Check out the last page of http://core.ecu.edu/psyc/wuenschk/docs30/CompareCorrCoeff.pdf

You should be able to convert cohen's d to a pearson r (find any meta-analysis calculator). Once that is done I don't see any reason you couldn't compare the two correlation coefficients using the procedures they describe. There is probably also a way to get at the same question by assessing model fit indices, but this might get dicey since your models aren't nested.

 

[Marissa4usa]

Sorry for the delay in my response. Turns out that Steiger's T-test was the way to go.
I have another question that is related. I am still running multilevel models and I am looking at the lagged effect of (1) variable A predicting variable B and (2) variable B predicting variable A. I'm predicting that A precedes B, but B doesn't precede A. In (1) I am controlling for the mean of variable A, in (2) I am controlling for the mean of variable B. (1) is significant, but (2) isn't - like I predicted. Now, I could stop here, given the difference in statistical significance. However, ideally, I'd like to go ahead and compare the effect sizes for these two models. Steiger's t-test doesn't work in this case as I would need a third variable. Is this something that's typically done or do people just report the p-values?

 

[CheetahGirl]

I may be way off...but seems like you could implement a structural equation modeling w/ all your variables (w/in 2 different models?!).

I've seen p-values reported against a visual model of the variables (looks nice) in dissertation defenses (but you may be asking for more...and my committee always says "report using APA manual"). So see how others have reported similar statistics & report like them. I'm sure you'll find some PubMed articles, if not on Google, for your Streigers t-test. Incidentally, if you end up comparing both models using SEM, I've definitely seen just p-values reported w/ the biggest stars & boldfaced highlights next to the significant relationships with values of "at least" p<.05. 🙂

Good luck whatever you do. Sounds cool.