Interaction not significant but Johnson-Neyman finding significant regions

[syzergy]

stats stats stats. I have a stats question. I posted this in the talkstats forum but I wanted to see if anyone here had thoughts. Any help is much appreciated.

I'm using the PROCESS macro and applying the Johnson-Neyman technique to some simple moderation analyses. I'm not quite sure what to make of some of my results. My results indicate the interaction between X and the moderator is not significant but the conditional effects of X on Y at -1SD, M, and +1SD values of my moderator are significant AND the Johnson-Neyman technique is identifying significant regions. How is that possible?

I can understand how the conditional effects at different values of the moderator could not be significant and still have the Johnson-Neyman pick up regions of significance (interaction could be significant at values outside -1SD or +1SD of the moderator). But how can the overall interaction not be significant?

I thought of a couple explanations... maybe the interaction is significant at really high or low values of my moderator but there's not enough people with these values. I checked and this isn't the case. One other possible reason: I'm including covariates in the model so maybe the interaction is not significant with the covariates but is significant without the covariates.

I welcome any thoughts. Should I interpret these interactions?

---

Members don't see this ad.

 

[syzergy]

I reran it without the covariates, same pattern of results.

 

[Therapist4Chnge]

One of my fav. shows.

I'm sorry I can't be of more assistance…hopefully Ollie or JS see your post and can help.

 

[syzergy]

No worries.

I figured out my moderator was actually a mediator for one of my IV-DV relationships but that's not the case for the other three models. Still have no idea what's going on with them.

I just want to get this freaking thesis done so I can get my master's in May and move on to cooler, better projects. I'm about to punch it in the face. Just let me write you without getting weird, thesis. Please.

 

[thewesternsky]

I'm not sure if I'm too young or just not American enough, but I don't recognize the show! T4C, what is it?

syzergy - I have nothing useful to say about your question; sorry. I do hope you get it figured out soon, though. Let us know what you find!

 

[OneNeuroDoctor]

The data must violate assumptions of homogeneity to use the J-N procedure. Therefore, the ANCOVA would not show significant interactions based on heterogeneity of slope, since the ANCOVA is only used when HOS is present. If there were significant interactions using ANCOVA, then the J-N procedure would not be needed as the data would have homogeneity of slope. The J-N procedure corrects the effect of heterogeneity of slope allowing for evaluation of regions where there is a significant interaction. I believe you are finding actual outliers at -/+ 1SD resulting in heterogenous slope and this is the reason J-N procedure is detecting significant interactions in regions with outliers.

You may need to get help from a research consultant to help run your data as this is a complicated procedure and most psychologists may need consult for doing J-N procedure.

 

[Therapist4Chnge]

I'm not sure if I'm too young or just not American enough, but I don't recognize the show! T4C, what is it?

It's from The IT Crowd, a British comedy series from a few years back.

 

[thewesternsky]

It's from The IT Crowd, a British comedy series from a few years back.

Oh, good-- So I'm just not British enough and/or aware of pop-culture enough. I've heard great things about that series. Thanks for the reply.

 

[Ollie123]

Sorry, missed this.

My experience has been that the talkstats crowd is relatively useless unless you are asking what "median" means or have a complicated question about the underlying mathematical principles. Applied questions rarely seem to be answered there.

RE: your question - do you have a main effect of X? I'm a bit outside my element since I don't think I've even once seen Johnson-Neyman used in my field (not that its inappropriate - just that its not the culture for whatever reason). I'm also not entirely sure I have enough information to answer the question without seeing more of the data. That said - there is a reason you are generally only supposed to follow up in that way after you find a significant interaction. My gut says you have a main effect of X on Y. I don't know how well this will be communicated in a post but picture the below as a graph.

--------------------------


--------------------------

Top line X = 1, Bottom line X = 0, moderator on the X axis, DV on the Y axis. Doesn't matter if your IV is continuous and not dichotomous, the idea is the same.

Assuming the separation is large enough, you would observe significant differences for any point along the moderator that you examine. The idea behind following up is that with different slopes the lines might not differ at certain points along the moderator. Like I mentioned above, I don't know much about the Johnson-Neyman technique but my 30 second googling implied the principle should be the same. Do a simple slopes or better yet just plot it. If your lines look roughly parallel, I'd wager I'm right and you are picking up on the main effect.

 

[EmotRegulation]

If your moderation isn't significant, and you are still seeing significant effects in the "conditional effects of X on Y" section, that means you have a main effect of X on Y, but no moderation. The conditional effects will show you if M moderates the effect, by looking at the effects of X on Y at -1SD, 0SD and +1SD of the moderator. In a significant moderation, it's likely that one of these will be NOT significant. It is of course possible to have a significant moderation with all three conditional effects still significant, you'd just be finding that the slopes of X on Y differ significantly at the sampled levels of M.

You are best off ignoring the Johnson-Neyman results, and just plot it instead (/plot = 1) as Ollie suggested.....I'd bet about anything that you have a significant effect of X on Y and that's it.

I pretty much spend my life running PROCESS models these days, and the output can get tricky if you're not used to it!

 

[syzergy]

Sorry I missed all these replies! I subscribed to the thread but SDN stopped emailing me updates. Thank you all for your responses. I met with one of my readers to see what was up.

He said PROCESS gives you the simple effects and JN results automatically even if your interaction is not significant. If I was doing the moderation with the typical SPSS procedures, I would see there was no interaction and wouldn't even probe. He said to just ignore the conditional effects and JN results. I can still plot them but they're not significant so it doesn't really matter.

Thanks for your responses, research gods. They pretty much confirm what I was told. Sorry I didn't see them earlier!

 

[EmotRegulation]

PROCESS won't give you the JN results unless you ask for them....so you could take that out of your syntax. (note: I know you can "hard install" PROCESS and use it from the menu system, but I have not experimented with that much because syntax is much easier).

 

[aquarius 71]

Hi participants. I just joined this forum and found your answers to be particularly helpful as I am also using PROCESS MACRO for my dissertation. Please help me out with this . As I run my moderation results, with a significant positive interaction, the region (mean +/_ 1 SD) shows that the interaction is significant for both values (MEAN+/- 1 SD ) with a negative beta for (mean -1 SD) and positive beta for (mean + 1 SD). Is it possible to have significant betas for both low and high values?How should I interpret them?

 

[EmotRegulation]

Make a plot!!! The PROCESS output gives you the data points for a plot if you ask for it (/plot = 1). You'll have to actually graph it yourself but you can do that in Word or Excel. It will basically give you the low, mod and high values of X predicting Y at different levels of W (the moderator is called W in PROCESS 3.0, and I use that convention to separate from M, which is a mediator).

In short, sure it's possible to have significant betas for both low and high values. That suggests you have a crossover interaction--at -1SD level of the moderator the relationship between X and Y is negative, and at +1SD level of the moderator the relationship between X and Y is positive.

But really, it will just make much more sense if you graph it.