There are two oft-cited assumptions for Analysis of Covariance (ANCOVA), which is used to assess the effect of a categorical independent variable on a numerical dependent variable while controlling for a numerical covariate:
1. The independent variable and the covariate are independent of each other.
2. There is no interaction between independent variable and the covariate.
In a previous post, I showed a detailed example for an observational study where the first assumption is irrelevant, but I have gotten a number of questions about the second.
So what does it mean, and what should you do, if you find an interaction between the categorical IV and the continuous covariate?
Once again, rather than focus on the “rules”, think about what an interaction tells you about the effects of the independent variable, and how you can best communicate the results.
In this (real) example, adult learners were assigned to one of four educational conditions: a control and three experimental conditions. The outcome variable is satisfaction with their training experience and each condition implemented a specific learning strategy. Participants’ age was used as a covariate, as previous research indicated that it is related to satisfaction.
Here is a graph of the actual results. The colored lines indicate the association between age and satisfaction within each of the four experimental groups. The purple line is the control group.
As they’re regression lines, the height of each line at a specific age can be interpreted as the mean value of satisfaction at that age for that specific group. So, for example, you can see that the mean satisfaction score is just about equal in all four groups at the point where the four lines cross, around age 25. (Incidentally, the fact that they all cross at the same age is a coincidence—this doesn’t have to happen).
The vertical black line at Age=25.8 represents the mean age of all participants. There were no significant differences in the mean age across groups, indicating that random assignment worked well.
It should be quite clear that the four lines are not parallel. This means that the relationship between age and satisfaction is not the same in every condition.
The p-value for the interaction between age and experimental group was .011. Quite significant.
Clearly, the ANCOVA assumption of parallel lines is not being met.
Okay, what does this mean for the analysis, and more importantly, the conclusions we draw about the data?
What the interaction says about the data
In the control condition and the green condition, age isn’t really related to satisfaction. Whether a participant is 20 or 50, the mean satisfaction is around 55. Yes, it’s slightly higher in the green group at higher ages, but by very little.
But in the blue and tan groups, there is an association between age and satisfaction. As a general pattern, young people tended to like their educational experience. Older people didn’t.
This effect is stronger in the tan group, but it’s there for both. There must have been something in those training programs that appealed more to young people.
This result is very interesting. If I am going to design a new training using one of these four strategies, and I know my audience is generally in their 20s, these results indicate that it really doesn’t matter which strategy I design it around. The mean satisfaction is pretty much equal for young participants.
But if my audience is in their 40s and 50s, the strategy is going to make a difference. If I use that tan strategy, my satisfaction ratings are likely to be pretty low. So I’ll probably get better ratings with the control strategy or the green one.
In other words, the effect of the learning strategy on satisfaction is different for different age participants.
Even if I didn’t hypothesize that, that’s an interesting and useful conclusion.
The parallel lines assumption
This is the point at which I usually get questions about what to do because a reviewer or committee member insists that ANCOVA can only be used if the lines are parallel.
This is true if the purpose of ANCOVA is to reduce error variation and allow us to report a single, overall effect of the independent variable on the dependent variable, at every value of the covariate.
It’s true that we can’t do that. We have already found that there is not a single effect of learning strategy treatment on satisfaction across all ages. We have to take age into account.
It doesn’t mean that random assignment didn’t work and it doesn’t mean that you should drop the covariate or the interaction term.
All it means is that you cannot:
1) Run the unique model without an interaction that many people mean when they say “ANCOVA.” We know that wouldn’t reflect the data.
2) describe the effect of the IV (training strategy) on the DV without also including some information about age.
Any stat software will allow you to get multiple-testing-adjusted p-values at specific values of a covariate. So it’s very easy to describe, and test, mean differences among the experimental groups at specific ages.
What analysis to do then? Your options:
1. Run just a one-way ANOVA, dropping the covariate.
But do this only if really, truly, the only thing you’re interested in is whether these experimental conditions show differences in satisfaction, regardless of age. The one-way ANOVA will reflect the effects of the treatment groups, regardless of age.
Of course, the main effect for condition in this full model with the interaction will test the same thing, as well as give you additional information at different ages. So your second option is:
2. Run the full model with the interaction, describe the results in detail, and don’t call it ANCOVA to appease reviewers.
I personally tend to advocate this approach, as it really tells the full story of what is going on with your data. But I concede that there are situations where the focus is very narrow and the full story is not of interest.
So whether it is correct to include the covariate and interaction in the model or to drop them both depends on what you really want to know about your data. They are answering different research questions.
But including the interaction term is not wrong because it doesn’t meet an assumption. The assumption is more about defining which models we can call ANCOVA than it is about which models best fit the data.