One of those “rules” about statistics you often hear is that you can’t interpret a main effect in the presence of an interaction.
Stats professors seem particularly good at drilling this into students’ brains.
Unfortunately, it’s not true.
At least not always.
This is one of those situations where I caution researchers to think about their data and what makes sense, rather than memorize a rule.
When interactions do make main effects nonsensical
Let me start with a situation in which the rule holds, so that you can see why it is important to consider.
Consider this simple interaction between two categorical predictors: Time and Condition.
At baseline (Time=0) there is no difference in the mean of outcome variable Y for the two groups.
I’m just going to assume that that difference is not statistically significant, because the means are right on top of each other. But in a real data output, you’d want to verify this with a simple effects tests.
But there is clearly an interaction here–there was a large change in the mean of Y for the treatment group, but not the control.
This is a perfect example of a case where both main effects will be significant, but they’re not meaningful.
For example, the main effect for Condition will compare the height of the red dotted line to the height of the black solid line. Assuming equal sample sizes , those heights will be compared halfway between time 0 and 1.
Again, assuming adequate power, that difference probably will be significant. But it’s not a meaningful main effect.
A main effect says that there is a difference between the group means, regardless of time.
Sure, if you average it out, that may be technically true. But it’s pretty clear that’s only true on average across the time points because of the big difference at Time 1. It’s not true at each time point. So to say that the means generally differ across conditions, regardless of time, isn’t really accurate.
This is sometimes referred to the interaction driving the main effects and this particular example is why your stat teacher doesn’t want you to blindly say that the significant main effect means anything.
When interactions don’t affect main effects
But of course, meaningful main effects can exist even in the presence of an interaction.
Take a very similar situation, with one important difference.
The interaction itself in the following graph is identical to the one above. The difference in means for the red triangles is once again 13 points, whereas the control group’s mean difference is negligible.
But the placement of the means is farther apart.
What makes the main effect meaningful here, despite the interaction, is that the treatment group’s mean is always higher than the control group’s. Even at Time=0.
That’s a meaningful main effect. It’s saying regardless of what is happening over time, the treatment group really does generally have higher means, regardless of when it’s measured.
Failing to mention this main effect, assuming it’s theoretically important, would be a shame. You’re missing the fact that the treatment group not only changed more, but started higher.
That’s a different situation than the first one, and you need to communicate that.
So when you have both a significant main effect and a significant interaction, stop and examine the results. Don’t assume the main effect is meaningless. It may be important.
Is there a statistical test that we could perform that would justify not interpreting a significant interaction, allowing us to interpret only the individual main effects?
Thanks!
I still don’t understand why the main effect is more important in the second example. In the second example, the main effect of the condition would be the average of the two simple effects of condition (assuming equal sample sizes). So, the main effect of the condition would be roughly (10+25/2), or 17.5. I get why one would emphasize that the two simple effects are in the same direction. If the simple effects are both significant, that would also be important to discuss. But, in the second example, the main effect is still an average of two significantly different effects. What is the point in reporting the effect of the condition when time=.5? Again, I understand the difference in interpreting the main effect when there is a “cross-over” and when there isn’t. For example, the main effect of the condition would be 0 and pretty useless if the lines looked like a perfect X. I just still do not understand why you would interpret the main effect of one variable unless the average value of the variable it is interacting with is of theoretical importance.
Previously I had not understood why in the presence of interactions the main effects aren’t interesting, and the first example is quite explanatory.
Anyway, now I think that main effects aren’t interesting even in the second example. There, Trt*1 > Trt*2 > Ctr*1 = Ctr*2. So the comparisons of the interactions already shows that all Trt > all Crt.
In fact, the main effects could be very interesting in the second example. They reveal that the two groups were different at baseline in regards to the outcome. Depending on the study design this could indicate a flawed randomization, or selection bias, among other things.
One never tests differences at baseline in randomized trials. It’s pure nonsense. Randomization ensures that all subjects come from a single population. So even, if the differences are statistically significant, it’s nothing but an artefact (it’s not impossible to sample very different observations into 2 groups from the same population). If the randomization if flawed, no test can show that. It’s procedural problem, not statistical. The opposite holds in non-randomized trials, where the differences at baseline come likely from two different populations. Then, the Lord’s paradox is almost certain.
Good afternoon,
I have always been taught that if a main/interaction effect is not sig then you should not use the post hoc pairwise comparisons even if significant. Hopkins suggests that planned contrasts of this kind are acceptable and we can use them. What is the ‘correct’ answer and which resources would help in understanding this matter?
Many thanks
Nice.