• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
The Analysis Factor

The Analysis Factor

Statistical Consulting, Resources, and Statistics Workshops for Researchers

  • Home
  • Our Programs
    • Membership
    • Online Workshops
    • Free Webinars
    • Consulting Services
  • About
    • Our Team
    • Our Core Values
    • Our Privacy Policy
    • Employment
    • Collaborate with Us
  • Statistical Resources
  • Contact
  • Blog
  • Login

When Does Repeated Measures ANOVA not work for Repeated Measures Data?

by Karen Grace-Martin 21 Comments

Repeated measures ANOVA is the approach most of us learned in stats classes for repeated measures and longitudinal data. It works very well in certain designs.

But it’s limited in what it can do. Sometimes trying to fit a data set into a repeated measures ANOVA requires too much data gymnastics. really.

These data gymnastics mean you’re throwing away good information and under-accounting for true variation among repetitions.

There are a few specific design and data situations that will eliminate repeated measures ANOVA as a reasonable approach.

Let’s go through seven of these and what the options are instead.

1. Missing Data on the outcome

One of the biggest problems with traditional repeated measures ANOVA is missing data on the response variable.

The problem is that repeated measures ANOVA treats each measurement as a separate variable. Because it uses listwise deletion, if one measurement is missing, the entire case gets dropped.

What to use instead: Marginal and mixed models treat each occasion as a different observation of the same variable. So you may lose the measurement with missing data, but not all other responses from the same subject.

 

2. Unbalanced number of repeats across individuals

A related problem is imbalance in the number of repeated responses from each individual.

This is common in observed data, where the number of repeats is uncontrollable. You measure a response each time some occurence happens.

Repeated measures ANOVA treats each response as a different variable. This causes two problems.

First, you will have a different number of response variables for each individual. If some have missing data in the last few responses, they’ll get dropped. (That dropping again. Ugh).

Second, the ANOVA will compare the responses to each other, assuming that each one represents a different condition. Here they don’t—they’re really interchangeable. But there is no way to turn off that comparison.

What to use instead: A mixed model can handle unequal repeats.

 

3. When time is continuous

In some repeated measures studies, each repeat occurs under a different experimental condition. There is a qualitative difference among the repeats. No problem here.

In others, the amount of time that has passed between repeats is important.  (Or equivalently, the amount of space if the repeats are say, along a transect). In other words, you want to treat the within-subjects effect of time as a continuous, quantitative variable.

This is theoretically valid and reasonable, but repeated measures ANOVA can only account for categorical repeats.

There are contrasts that allow you to order the categories and simulate a trend over time, but they’re not truly treating time as continuous.

What to use instead: A marginal or mixed model can treat time as a truly continuous effect.

 

4. Time-varying covariates

In some studies, the important predictor variables are measured on each repeat, right along with the response.

Because of that wide-data format, there’s no way to specify that each measurement of the covariate variable should only predict the corresponding response.

What to use instead: A marginal or mixed model can incorporate time-varying covariates.

5. Three (or more) level models

If the subjects themselves are not only measured multiple times, but also clustered into some other groups, you’ve got a three-level model.

For example, you may have students measured over time, but students are also clustered within classrooms.

Streams may be measured over time, but are also clustered into watersheds.

Patients measured over time are also clustered into medical centers.

In all these cases, the repeated measures ANOVA can account for the repeats over time, but not the clustering.

What to use instead: A mixed model can incorporate multiple levels.

 

6. Repeated measures across people and items

There is a repeated measures design that occurs in specific experimental studies. They’re common in linguistics and psychology. These are studies in which each subject is repeated measured across many trials. Each trial contains one item, and there are multiple items for each condition.

An example may be to measure reaction time of 50 participants to each of 20 high-frequency and 20 low frequency words.

It’s clear that the 40 reaction times are repeated across each participant, and we need to account for the fact that multiple responses from the same subject are correlated. After all, some participants will always be faster than others.

But each word also has 50 repeated measurements (one per participant) and those are also likely to be correlated to each other. Some words will elicit faster times than others, even within the same condition.

Repeated measures ANOVA can only account for the repeat across one type of subject.

What to use instead: A mixed model with crossed random effects.

 

7. Non-continuous outcomes

Finally, repeated measures ANOVA has assumptions of normality within each factor.

Sure, it’s robust to small departures of this assumption. And if the outcome variable is continuous, unbounded, and measured on an interval or ratio scale, you may be able to solve non-normality with a transformation.

But if you’ve got categorical outcomes or count outcomes, it’s not going to work. Luckily, there are other options.

What to use instead: A Generalized Estimation Equation (GEE) or Generalized Linear Mixed Model (GLMM).

 

Fixed and Random Factors in Mixed Models
One of the hardest parts of mixed models is understanding which factors to make fixed and which to make random. Learn the important criteria to help you decide.

Tagged With: crossed random effects, generalized linear mixed model, GLMM, linear mixed model, Marginal Model, Repeated Measures, repeated measures anova, time-varying covariates

Related Posts

  • Six Differences Between Repeated Measures ANOVA and Linear Mixed Models
  • An Example of Specifying Within-Subjects Factors in Repeated Measures
  • Linear Mixed Models for Missing Data in Pre-Post Studies
  • Why Mixed Models are Harder in Repeated Measures Designs: G-Side and R-Side Modeling

Reader Interactions

Comments

  1. Rebecca Bokoch says

    March 3, 2021 at 12:29 am

    If you are running a simple repeated measures ANOVA, looking at change overtime (time 1, time 2, time 3) for a measure of anxiety, and one of the measures of anxiety (time 1) is not normally distributed but the other two are (time 2 and time 3), do you:
    1) transform just anxiety at time 1?
    2) transform all measures of anxiety at time 1, 2, & 3?
    3) don’t transform any of the measures of anxiety

    If you do have to use a transformation, how do you interpret the results? Please let me know if you can advise. Thank you!

    Reply
    • Karen Grace-Martin says

      December 6, 2021 at 12:58 pm

      Rebecca,

      It won’t help to transform just time 1 because then your scaling will be totally off. And transforming all three time points might cause more problems than it solves. This might be one of those situations where there just isn’t a clear best way to do it. So you have to look at everything you’ve got and make the best decision you can. In this case, it would really depend on the way in which time 1 anxiety isn’t normal. Like, is it skewed right, bimodal, uniform?

      Reply
  2. Richard Anderson says

    July 7, 2020 at 2:38 pm

    I think there’s a problem with saying that repeated-measures ANOVA can’t handle the following:

    “5. Three (or more) level models
    If the subjects themselves are not only measured multiple times, but also clustered into some other groups, you’ve got a three-level model.”

    That’s just a repeated-measures ANOVA with with “X” as a within-subject factor (whichever repeated-measure “X” refers to) and “Class” as a between-subject factor.

    For example, you may have students measured over time, but students are also clustered within classrooms.

    Reply
  3. Sophia says

    February 22, 2020 at 8:05 pm

    Hi there, I have a question regarding repeated measures ANOVA. Does this test check for random effects in your data set? Does it tell you if there’s a special relationship between data points (e.g. subject 1 and subject 2 have similar values across different time points.) Does it check for if “when the values for subject 1 go up, the values for subject 2 go up too”?
    Thank you,
    Sophia

    Reply
  4. Roos says

    June 16, 2019 at 4:18 am

    Hi! I have to perform a repeated measure in SPSS and this works for the example that I found on YouTube. However, if I enter my own data, the outcome of the Mauchly’s Test of Sphericity is a . by significance, a 0 by df, and 1,000 for almost all other values (except for approx. chi-square which is ,000). Could someone help me, because I do not know what I did wrong?

    Reply
    • Karen Grace-Martin says

      August 22, 2019 at 11:59 am

      Hi Roos,

      It’s too hard to tell from your description what happened. I can tell you Mauchley isn’t a very good test of sphericity anyway, but it’s strange you didn’t even get a value.

      Reply
    • mohammed says

      February 20, 2020 at 12:49 pm

      if the repeats number is two, you will not have results for this test. you need at least 3 repeats.

      Reply
  5. Ray says

    May 1, 2019 at 1:36 pm

    Hi. I have a design where participants view images repeatedly, and the images have 3 levels. I have a continuous predictor (i.e., scale measuring life history). Can a mixed model (LME) be appropriate for this type of design? Thank you.

    Reply
    • Karen Grace-Martin says

      May 9, 2019 at 9:49 am

      Hi Ray,

      It sounds like it, but I would need to know a lot more detail before I could give you accurate advice about the analysis to take for any given study.

      Reply
  6. William says

    June 19, 2018 at 3:57 am

    Thanks! This REALLY helped!!

    Reply
  7. Lena says

    June 14, 2018 at 9:17 am

    Hi Karen,

    Thank you very much for providing this info.

    I would be thankful if you could provide me a quick feedback regarding the best analysis for my situation. I have one sample which went through physical activity program. We measured participants at baseline, 10 and 20-week follow up (no control condition). My retention data is very poor Baseline= 58 participants, 10-weeks= 39 and 20-weeks= 21. Our outcomes are a few tests on a continuos scale (time, repetitions).

    I would be thankful for your tip.

    Greetings,
    Lena

    Reply
    • Karen Grace-Martin says

      October 26, 2018 at 5:18 pm

      Hi Lena,

      Unfortunately, there isn’t a clear answer. It really depends on why people are dropping out and how much you can assume randomness of dropout.

      Reply
  8. Faraz Farooqi says

    February 21, 2017 at 8:12 am

    Good Explanation. I have a question we record values on TWO occasions with same participant every time. Can we run repeated measure of ANOVA or I should go for Paired Sample t-test ( non-parametric )?
    My variables are continuous in nature and my data is not distributed normally.

    Reply
  9. Amanda says

    February 15, 2017 at 5:02 am

    Karen,

    Thanks for your explanation.

    Currently, I am running an experiment with 5 independent variables and two dependent variables (response time and correctness). The correctness is a binary response, and I used GEE. Also, since there are missing data for the response time, I used Mixed Model. However, when I found a significant result for one independent variable which has three levels, I would like to do multiple comparisons to know where the significant results come from. Is there any suggestion you could provide to do this multiple comparisons? Or should I look into the Estimates of Fixed Effects table?

    Thanks.

    Best,

    Amanda

    Reply
  10. Erin says

    July 10, 2015 at 4:38 pm

    Karen,

    I really enjoyed readying this. I was struck by a line in your first paragraph “Sometimes trying to fit a data set into a repeated measures ANOVA requires too much data gymnastics—averaging across repetitions or pretending a continuous predictor isn’t”.

    This is the situation I am currently in. I am running a repeated measures in SPSS, and I have two predictor variables, one is continuous (which I entered as a co-variate) and the other categorical (which I entered as a between-subject factor). I want these two predictor variables to interact, so what I did was alter the syntax and added the interaction in the “design” line. The interaction is significant, and now I am trying to interpret the interaction. I can split the file by the categorical predictor and determine the level of the categorical variable that is moderated by my continuous predictor. Because GML does not produce a beta coefficient, I am having a hard time knowing the direction of this association.

    I feel like I might be missing something obvious. Any thoughts you have are greatly appreciated.

    Erin

    Reply
    • Karen says

      July 14, 2015 at 9:43 am

      Hi Erin,

      I think what you’re missing is that GLM does produce a beta coefficient. You need to use /Print parameterSolution.

      However, in Repeated Measures GLM, it may not be what you want. I suspect you’ll have to use Mixed instead of RM Anova.

      You may find this helpful: https://www.theanalysisfactor.com/resources/mixed-multilevel-models/

      Reply
      • Gerard says

        May 15, 2020 at 9:41 am

        Hi Karen,

        Great website. I’m wondering as well how to use the /PRINT SOLUTION command in the GLM function in SPSS to get a beta coefficient. The only outputs I have are F statistics & the p-value for each co-variate.
        Whenever I try to type /PRINT=SOLUTION or SOLUTION into the syntax it generates an error. It seems /PRINT SOLUTION is a ‘mixed’ syntax?
        At my wits end at the moment!

        Many thanks!

        Reply
        • Karen Grace-Martin says

          June 2, 2020 at 4:25 pm

          Hi Gerard,

          Oh, I think you’re right. Sorry about that. Try /print parameter in GLM. (I’ll fix that).

          One nice thing about SPSS is if you can type the first letter of an option, it will give you a drop down menu of all the possible options. So if one isn’t working, you can see what does.

          Reply
  11. Mike Lyell says

    October 22, 2014 at 7:30 am

    Beautiful explanation. However, I´m trying to analyze a dataset and predict a binary dependent variable measured once several years after obtaining multiple measurements on a independent variable (time-varying covariate) in an unbalanced design. As the dependent variable is only measured once I´m uncertain as to the correct approach in analyzing this. I have previously done cox analyses with time-varying covariates, but I´ve never seen an approach with time-varying covariates for logistic regression. Any ideas?

    Reply
    • Karen says

      October 23, 2014 at 2:01 pm

      Hi Mike,

      There are a few options, but the most common would be to summarize the time-varying covariate with something like it’s max, mean, slope of its change over time and use that as a predictor. If there aren’t too many time points of this variable, you can also use each value as a covariate.

      Reply
  12. Craig Marsden says

    September 24, 2014 at 3:06 pm

    Really well structured explanation.

    Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Please note that, due to the large number of comments submitted, any questions on problems related to a personal study/project will not be answered. We suggest joining Statistically Speaking, where you have access to a private forum and more resources 24/7.

Primary Sidebar

This Month’s Statistically Speaking Live Training

  • Member Training: Introduction to SPSS Software Tutorial

Upcoming Free Webinars

Poisson and Negative Binomial Regression Models for Count Data

Upcoming Workshops

  • Analyzing Count Data: Poisson, Negative Binomial, and Other Essential Models (Jul 2022)
  • Introduction to Generalized Linear Mixed Models (Jul 2022)

Copyright © 2008–2022 The Analysis Factor, LLC. All rights reserved.
877-272-8096   Contact Us

The Analysis Factor uses cookies to ensure that we give you the best experience of our website. If you continue we assume that you consent to receive cookies on all websites from The Analysis Factor.
Continue Privacy Policy
Privacy & Cookies Policy

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
Non-necessary
Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.
SAVE & ACCEPT