As mixed models are becoming more widespread, there is a lot of confusion about when to use these more flexible but complicated models and when to use the much simpler and easier-to-understand repeated measures ANOVA.
One thing that makes the decision harder is sometimes the results are exactly the same from the two models and sometimes the results are vastly different.
In many ways, repeated measures ANOVA is antiquated — it’s never better or more accurate than mixed models. That said, it’s a lot simpler.
As a general rule, you should use the simplest analysis that gives accurate results and answers the research question. I almost never use repeated measures ANOVA in practice, because it’s rare to find an analysis where the flexibility of mixed models isn’t an advantage in either giving accurate results or answering a more sophisticated research question.
But they do exist.
Here are some guidelines on similarities and differences:
1. Simple design, complete data, normal residuals
If the design is very simple and there are no missing data, you will very likely get identical results from Repeated Measures ANOVA and a Linear Mixed Model. By simple, I mean something like a pre-post design (with only two repeats) or an experiment with one between-subjects factor and another within-subjects factor.
If that’s the case, Repeated Measures ANOVA is usually fine.
The flexibility of mixed models becomes more advantageous the more complicated the design.
2. Non-normal residuals
Both Repeated Measures ANOVA and Linear Mixed Models assume that the dependent variable is continuous, unbounded, and measured on an interval or ratio scale and that residuals are normally distributed.
There are, however, generalized linear mixed models that work for other types of dependent variables: categorical, ordinal, discrete counts, etc. So if you have one of these outcomes, ANOVA is not an option. There is no Repeated Measures ANOVA equivalent for count or logistic regression models.
(There are GEE models, but they are closer in many ways to mixed in terms of setting up data, estimation, and how you measure model fit. You can’t calculate sums of squares by hand, for example, the way you can in Repeated Measures ANOVA).
3. Clustering
In many designs, there is a repeated measure over time (or space), but subjects are also clustered in some other grouping. Students within classroom, patients within hospital, plants within ponds, streams within watersheds, are all common examples.
A repeated measures ANOVA can’t incorporate this extra clustering of subjects in some other clustering, but mixed models can. (In fact, this kind of clustering can get quite complicated.)
4. Missing Data
As implied above, mixed models do a much better job of handling missing data. Repeated measures ANOVA can only use listwise deletion, which can cause bias and reduce power substantially. So use repeated measures only when missing data is minimal.
5. Time as Continuous
Repeated measures ANOVA can only treat a repeat as a categorical factor. In other words, if measurements are made repeatedly over time and you want to treat time as continuous, you can’t do that in Repeated Measures ANOVA.
For example, let’s say you’re measuring anxiety level during weeks 1, 2, 4, 8, and 16 of an anxiety-reduction intervention. In mixed models you have the choice to treat those 5 time points as either 5 discrete categories or as true numbers, which accounts for the different spacing of the weeks.
Repeated Measures ANOVA can only do the former.
6. Differing number of repeats
Repeated measures ANOVA falls apart when repeats are unbalanced, which is very common in observed data.
A common study is to record some repeated behavior for individuals, then compare some aspect of that behavior under different conditions.
I’ve seen this kind of study in many fields. One compared the diameter of four species of oak trees at shoulder height in areas that were and were not exposed to an invasive pest. Because those trees were observed, not planted, there was a different number of each species in each plot. So once again, some plots had many repeated data points for each species, while others had only a few.
Repeated measures ANOVA can’t incorporate the fact that each plot has a different number of each type of species. It can only use one measurement for each type.
The traditional way of dealing with this is to average multiple measures for each type, so that each infant and each plot has one averaged value for each breath type/species. The problem with this is it under-represents the true variability in the data (this is bad). Those averages aren’t real data points — they’re averages with variability around them.
Mixed models can account for this variability and the imbalance with no problems.
Conclusion
There are other differences, of course, but some of those get quite involved.
So what it really comes down to is Repeated Measures ANOVA is a fine tool for some very specific situations. Once you deviate from those, trying to use it is like sticking that square peg through the round hole.
You might get it through, but you’ll mangle your peg in the process.
