When Does Repeated Measures ANOVA not work for Repeated Measures Data?

Repeated measures ANOVA is the approach most of us learned in stats classes for repeated measures and longitudinal data. It works very well in certain designs.

But it’s limited in what it can do. Sometimes trying to fit a data set into a repeated measures ANOVA requires too much data gymnastics. really.

These data gymnastics mean you’re throwing away good information and under-accounting for true variation among repetitions.

There are a few specific design and data situations that will eliminate repeated measures ANOVA as a reasonable approach.

Let’s go through seven of these and what the options are instead.

1. Missing Data on the outcome

One of the biggest problems with traditional repeated measures ANOVA is missing data on the response variable.

The problem is that repeated measures ANOVA treats each measurement as a separate variable. Because it uses listwise deletion, if one measurement is missing, the entire case gets dropped.

What to use instead: Marginal and mixed models treat each occasion as a different observation of the same variable. So you may lose the measurement with missing data, but not all other responses from the same subject.

2. Unbalanced number of repeats across individuals

A related problem is imbalance in the number of repeated responses from each individual.

This is common in observed data, where the number of repeats is uncontrollable. You measure a response each time some occurence happens.

Repeated measures ANOVA treats each response as a different variable. This causes two problems.

First, you will have a different number of response variables for each individual. If some have missing data in the last few responses, they’ll get dropped. (That dropping again. Ugh).

Second, the ANOVA will compare the responses to each other, assuming that each one represents a different condition. Here they don’t—they’re really interchangeable. But there is no way to turn off that comparison.

What to use instead: A mixed model can handle unequal repeats.

3. When time is continuous

In some repeated measures studies, each repeat occurs under a different experimental condition. There is a qualitative difference among the repeats. No problem here.

In others, the amount of time that has passed between repeats is important.  (Or equivalently, the amount of space if the repeats are say, along a transect). In other words, you want to treat the within-subjects effect of time as a continuous, quantitative variable.

This is theoretically valid and reasonable, but repeated measures ANOVA can only account for categorical repeats.

There are contrasts that allow you to order the categories and simulate a trend over time, but they’re not truly treating time as continuous.

What to use instead: A marginal or mixed model can treat time as a truly continuous effect.

4. Time-varying covariates

In some studies, the important predictor variables are measured on each repeat, right along with the response.

Because of that wide-data format, there’s no way to specify that each measurement of the covariate variable should only predict the corresponding response.

What to use instead: A marginal or mixed model can incorporate time-varying covariates.

5. Three (or more) level models

If the subjects themselves are not only measured multiple times, but also clustered into some other groups, you’ve got a three-level model.

For example, you may have students measured over time, but students are also clustered within classrooms.

Streams may be measured over time, but are also clustered into watersheds.

Patients measured over time are also clustered into medical centers.

In all these cases, the repeated measures ANOVA can account for the repeats over time, but not the clustering.

What to use instead: A mixed model can incorporate multiple levels.

6. Repeated measures across people and items

There is a repeated measures design that occurs in specific experimental studies. They’re common in linguistics and psychology. These are studies in which each subject is repeated measured across many trials. Each trial contains one item, and there are multiple items for each condition.

An example may be to measure reaction time of 50 participants to each of 20 high-frequency and 20 low frequency words.

It’s clear that the 40 reaction times are repeated across each participant, and we need to account for the fact that multiple responses from the same subject are correlated. After all, some participants will always be faster than others.

But each word also has 50 repeated measurements (one per participant) and those are also likely to be correlated to each other. Some words will elicit faster times than others, even within the same condition.

Repeated measures ANOVA can only account for the repeat across one type of subject.

What to use instead: A mixed model with crossed random effects.

7. Non-continuous outcomes

Finally, repeated measures ANOVA has assumptions of normality within each factor.

Sure, it’s robust to small departures of this assumption. And if the outcome variable is continuous, unbounded, and measured on an interval or ratio scale, you may be able to solve non-normality with a transformation.

But if you’ve got categorical outcomes or count outcomes, it’s not going to work. Luckily, there are other options.

What to use instead: A Generalized Estimation Equation (GEE) or Generalized Linear Mixed Model (GLMM).

Fixed and Random Factors in Mixed Models

One of the hardest parts of mixed models is understanding which factors to make fixed and which to make random. Learn the important criteria to help you decide.

Take Me to The Video!

Tagged as: crossed random effects, generalized linear mixed model, GLMM, linear mixed model, marginal models, Repeated Measures, repeated measures anova, time-varying covariates