Not too long ago, I received a call from a distressed client. Let’s call her Nancy.

Nancy had asked for advice about how to run a repeated measures analysis. The advisor told Nancy that actually, a repeated measures analysis was inappropriate for her data.

Nancy was sure repeated measures *was *appropriate and the response led her to fear that she had grossly misunderstood a very basic tenet in her statistical training.

**The Design**

Nancy had measured a response variable at two time points for two groups: an intervention group, who received a treatment, and a control group, who did not.

Both groups were measured before and after the intervention.

**The Analysis**

Nancy was sure that this was a classic repeated measures experiment with one between subjects factor (treatment group) and one within-subjects factor (time).

The advisor insisted that this was a classic pre-post design, and that the way to analyze pre-post designs is not with a repeated measures ANOVA, but with an ANCOVA.

In ANCOVA, the dependent variable is the post-test measure. The pre-test measure is not an outcome, but a covariate. This model assesses the differences in the post-test means after accounting for pre-test values.

The advisor said repeated measures ANOVA is only appropriate if the outcome is measured multiple times after the intervention. The more she insisted repeated measures didn’t work in Nancy’s design, the more confused Nancy got.

**The Research Question**

This kind of situation happens all the time, in which a colleague, a reviewer, or a statistical consultant insists that you need to do the analysis differently. Sometimes they’re right, but sometimes, as was true here, *the two analyses answer different research questions*.

Nancy’s research question was whether the mean change in the outcome from pre to post differed in the two groups.

This is directly measured by the time*group interaction term in the repeated measures ANOVA.

The ANCOVA approach answers a different research question: whether the post-test means, adjusted for pre-test scores, differ between the two groups.

In the ANCOVA approach, the whole focus is on whether one group has a higher mean after the treatment. It’s appropriate when the research question is not about gains, growth, or changes.

The adjustment for the pre-test score in ANCOVA has two benefits. One is to make sure that any post-test differences truly result from the treatment, and aren’t some left-over effect of (usually random) pre-test differences between the groups.

The other is to account for variation around the post-test means that comes from the variation in where the patients started at pretest.

So when the research question is about the difference in means at post-test, this is a great option. It’s very common in medical studies because the focus there is about the size of the effect of the treatment.

**The Resolution**

As it turned out, the right analysis to accommodate Nancy’s design *and* answer her research question *was* the Repeated Measures ANOVA. (For the record, linear mixed models also work, and had some advantages, but in this design, the results are identical).

The person she’d asked for advice was in a medical field, and had been trained on the ANCOVA approach.

Either approach works well in specific situation. The one thing that doesn’t is to combine the two approaches.

I’ve started to see situations, particularly when there is more than one post-test measurement, where data analysts attempt to use the baseline pre-test score as both a covariate and the first outcome measure in a repeated measures analysis.

That doesn’t work, because both approaches remove subject-specific variation, so it tries to remove that variation twice.

**Analyzing Repeated Measures Data: GLM and Mixed Models Approaches**.

{ 65 comments… read them below or add one }

Here are two article related to this question, I hope all of you can read and discuss here. Because it is a interesting question recently bother me.

Paper 1. Dimitrov, Dimiter M., and Phillip D. Rumrill Jr. “Pretest-posttest designs and measurement of change.” Work 20.2 (2003): 159-165.

This paper says: “….and one within-subjects (pretest-posttest) factor.

Unfortunately,this is not a healthy practice because previous

research [10,12] has demonstrated that the results

provided by repeated measures ANOVA for pretestposttest

data can be misleading. Specifically, the F test

for the treatment main effect (which is of primary interest)

is very conservative because the pretest scores are

not affected by the treatment. A very little known fact

is also that the F statistic for the interaction between the

treatment factor and the pretest-posttest factor is identical

to the F statistic for the treatment main effect with

a one-way ANOVA on gain scores [10]. Thus, when

using repeated measures ANOVA with pretest-posttest

data, the interaction F ratio, not the main effect F ratio,

should be used for testing the treatment main effect.

A better practice is to directly use one-way ANOVA

on gain scores or, even better, use ANCOVA with the

pretest scores as a covariate.”

And this website:

http://www.jerrydallal.com/lhsp/prepost.htm

Actually, if you use gain score (post – pre) as a outcome to conduct ANCOVA with pre score as a covariate, the p value of between-subject effect (treatment group) is the same as using only post score as a outcome.

post – pre = intercept + prescore + treatment + error

is equivalent with

post = intercept + prescore + treatment + error

for testing treatment effect

So if I use the gain score in the model, do I answer the question:

“whether the mean change in the outcome from pre to post differed in the two groups.(we need to add conditioned on the same mean pre score because prescore was used as covariate”. If yes, then can we use ANCOVA instead of mixed model to test interaction term? Which is better?

this seems like an old post, but hopefully i can still get a response.

I have a very similar case as described here and am indeed planning to do a repeated measures ANOVA with one between subjects factor (treatment group) and one within-subjects factor (time). Can you give a hint how to set this up with linear regression? I have unequal sample sized in the groups, and I also might need to ad a third factor. Plus, I’m just curious.

Hi.

This is my topic: the effect of strategy based instruction on learner reading autonomy, reading performance, & reading motivation in tertiary education in Iran. Which kind of test is appropriate for that? I have 4 variables, strategy based instruction is independent and the rest of them are dependent. How show should I do test?

Any help will be appreciated.

Hi Karen! I am as crazily confused as the others here. But I hope you could look at my dilemma. I don’t know what statistical tool to use to test my hypotheses. I’m having one group with more than a pair of observations. I’m testing a group of 15 athletic swimmers and will subject them to blood gas analysis to get the difference of their arterial oxygenation. I will have a blood gas baseline measurement (before swim), blood gas results at 15 minute interval swim (post result), and blood gas results at 30 minute interval swim (another post result, but will be compared to the previous post result instead of comparing it to the baseline measurement (before swim). Please help me. I dont know what to do anymore. Thanks.

Hi Karen, I feel like Nancy since I am not sure which method to apply. I am applying one pre and one post tests (identical). I have control and experiment groups and working with three school. In summary, three school, one control and one experiment class in each school. One and one post test for each. I am measuring the influence of an educational intervention. Is this a repeated measure ANOVA?

I will apply Levene and t-test to check the homogeneity of variance and means at the beginning.

Hi Karen, I have a question here. Recently I conducted a school based health education on dengue. I used pre- and post-test design whereby an education material was provided during the inter-phase of the study. Focus is to evaluate the intervention (to see gain/changes in knowledge, attitude and practices). Usual analysis method for this kind of data in SPSS is Dependent-t-test, but it only applies if the data are paired. In my case, I did not paired the data (the respondents) during the pre and post-test. Also, I have unequal sample size. Do you have any suggestions on what kind of test I can use in SPSS to analyse this data? TQVM!

Hi Kareen,

I have a question.I am conducting a research study and I want to know wether the intervention leads in that clients participate more in group treatment or not? and then compare their session attendance with previous clients referred to the same center and clients referred to another center for the same problem( there are some problems that we can not have control groups). I also want to check some secondary outcomes in the group who received MI in three different time which is pre, post intervention, and post group treatment. I thought that it’s better to use repeated measure anova, after discussion with a statistician she told me to use ANCOVA for all of my research questions and outcomes. I am a bit confused. Would you please let me know which test should I use?

Hi Karen

I have carried out a mindfulness intervention with elements of qualitative a bit of a quant. I collected data pre and post intervention with 2 groups, but no control. What I did not do was pair the data (!).

I am looking to see if there are any differences in stress and levels of mindfulness pre and post intervention. Is there still an analysis that may work?

Hey Dee,

I am actually doing a very similar study with mindfulness. I would love to hear what you have done since you last asked your questions. Are you useing scales and comparing the answers on those scales from pre to post?

Best,

Francisco

I am looking at 20 companies which were applying a particular set of accounting standards in their financial reporting. After 5 years of application, they were required to change the standards and apply a different set of standards. I want to analyze the effect of the new standards on the companies’ financial performance and market value, which statistical test will be more appropriate?

Hi. I am studying a QoL questionaire before surgery and 1 month after surgery. I donot have any control group. How do i analyse results? Further if i postulate that after surgery the QoL improves by 20% than what should be my sample size?

please i want to use ancova to analyse data but i dont know how to go about it. I dont know independent, covariance and factor

Hi,

First, let me express my gratitude for the useful information provided here.

Then, I should say that I have a similar problem: I had a homogeneous sample (selected via a homogeneity test), then randomly divided them into two groups. After that, I administered a pretest (writing skills).Then, 20 sessions of treatment (in control group: I taught academic writing and in the treatment group: academic writing plus the treatment). Next, I gave them a post-test to calculate the (probable) significance of the treatment. Now, should I be using ANCOVA for the analysis or just t-test?

Please help π

Thanks a lot.

Hi Karen,

I have 2 groups, 1 group that received an intervention (170 participants) and another that didn’t receive the intervention (30 participants). I am trying to assess whether the intervention has 1) an impact on 3 process variables and 2) 4 DVs.

There was no chance to randomly allocate the groups… they were 2 student cohorts measured at equivalent times in their training… one going into winter and 1 going into summer.

Problems (1) – controlling for baseline differences, (2) dealing with unequal cell sizes and (3) dealing with seasonal effects.

Given that I am looking at a total of 7 continuous variables…

I have a few questions:

1) should I do a repeated measures MANCOVA?

2) If I do repeated measures MANCOVA will this control for baseline differences?

3) I have been advised that I could do a matched subjects design…..draw a matched sample of 30 from the 170…. but others say that this reduces power.

Hi Karen,

I am completing a repeated measures ANCOVA, with one covariate and between subjects factor. I am a bit confused as to how to check the homogeneity of regression slopes assumption. Can you shed any light on this?

Any help will be greatly appreciated!

Alice Maier

The topic is “Effect of Pilates and Swiss ball training on balance, strength and sleep quality”. Herein are attached tables of the data collection. We intend to apply ANCOVA for the research question.

Questions we are searching answer for are:

Improvement in all three variables in the Pilates group,

Improvement in all three variables in the Swiss ball group,

And the final inference is to be drawn whether which group has shown better improvement on comparison for the three variables.

Please suggest which table (pre/pre, post/post or pre/post, pre/post) should be used for the statistical analysis using ANCOVA test.

Hi,

I am doing a simple pretest, education, posttest. What I want to know is if the education made a difference for the group. The group was made up of 8 with 12 questions. I tried the t-test and was told I had to have 30 participants. I explained my results and I am still told I need help explaining and the t-test was not the test to use. So, I have been reviewing the ANOVA is that what I would use. And how do I explain the difference?

Thanks, M

Hi, I really need some help with this. I am investigating whether there is any change on maternal strategies before and after an intervention.I am coding on materlan strategies in response to children’s errors. I have coded the variables pre-test and at post-test. I have divided the variables with the total number of errors at pre-test and at post test to adjust fr the scores and have a fair comparison. I have run a 2X2 repeated measures ANOVA but now I have second thoughts. Would it be wiser to run an ANCOVA and covary for the number of errors???

We have conducted an efficacy study of an intervention Program. 55 participants completed the Program and 37 took part in a control group. At baseline no diferences were found between the groups regarding the study variables or demographic variables. We used repeated measures ANOVA. We are now interested in understanding the role of some variables, for example experiential avoidance, in the decrease that occured in depressive symptoms from T1 ( baseline) to T2 (post treatment). What would you suggest?

i have 4 experimental group, each experimental group was measured as pre, mid and post score for all the groups. what type of statistics i want to use

Dear Karen,

Your answers are clear and good. Am a doctoral student and my objective of the study is the use of SMS for learning among undergraduate students

of Visual Communication and considers pre-post test design. My doubt is, is it necessary to have same questions to conduct the study. For a period of month the students will be receiving important terms, glossaries of communication.

Hi Karen,

I have almost the same design. I am trying to evaluate the effectiveness of intervention on specific performance outcomes in different conditions and I have control group. I have 2 conditions and both conditions have 2 levels( I ll use mixed anova). I am really confused about should I take pre-post test as a factor into analyses or should I get the differences between pre and post. what do u think?

Hi Karen, I am a big fan of your site. I have a related question. I have an experiment with 2 fixed factors and 1 random factor of site. The treatments were assigned randomly at each site and each combination has 4 plots. I am interested in the effects of the fixed factors (inl. interactions) on the post-treatment grass shoot counts.

Since I have a random factor and dealing with counts I have chosen the SPSS GENLINMIXED procedure with a Poisson distribution. However, the resulting model is over-dispersed. Before turning to the negative binomial, I would like to include a continuous covariate (grass shoot counts in each plot pre-treatment) to try to explain more of the variance. Is this a right move?

Also, the GENLINMIXED model seems to be taking the covariate as a fixed factor. How would I interpret the results in this case?

Many thanks,

Jo

Hi Karen,

Thank you for your post. it seems that sometimes itβs hard to know what is exactly the right analysis. In my case, an educational intervention was implemented and pre/post data survey was admistered to evaluate effectiveness on health outcomes. I would like to also know if the patients who indicated that they had arthristis expereinced reduced level of pain in pre/post intervention.

well your answer made me much more confused lol.These are my research questions:

1 What are the different effects of text-picture and audio-picture in facilitating EFL vocabulary immediate recall?

2 What are the different effects of text- picture and audio-picture in facilitaing EFL reading comprehesion? I would really appreciare tour help Karen.Thank you in davance

hi, im really confused about the kind of measueremnt i should use:ANCOVA, or repeated measure ANOVA.I would like to conduct a pretest-postest design. I would like to investigate the effects of multimedia on the vocabulary acquisition and reading comprehension. The two experimental groups will receive a pretest and then a postest for both dependent variables reading comprehension and vocabulary acquisition.

Hi Asma,

Based on what you’ve told me, it could be either. It depends on what you’re interested in testing: how much the multimedia

changesvocabulary and reading comprehension OR whether the final vocabulary and reading comprehension is higher based on which multimedia exposure they got, controlling for where each person started. They’re different research questions. One is about change/growth/development. The other is about who is better off at the end.Hi karen,

I have benefited a lot from your explanation.

I have a pre-post design, and it is a uncontrolled prospective study. We want to compare whether the seurm BDNF in patients after 8 weeks treatment is higher than those in patients before treatment . in this situation, can I use repeated measures ANOVA? and if I need to adjust for

potential confounder of factors such as duration of illness, severity before treatment ? if so, how can I adjust these confounders?

Thank you so much, and looking forword to your response!

Hi Shenhe,

Yes, you can use a repeated measures approach. I suspect, though, that you’ll be better off with a marginal model, in order to incorporate those covariates. RM anova can include covariates, but it’s limited. See this article on approaches to repeated measures.

Dear Karen,

Thanks for your answer, and it benefits a lot. However, I still have a question. In our study, there is only one antidepressant treatment patients group,and there is not a placebo treatment patients group.

my question is that, in this situation, how can I calculate the connection between the BDNF decrease and the course of treatment, especially when there were many confunding factors such as age, sex, severity before treatment?

In general, it seems that adjusting the covariates need two groups, and since our study lacked of placebo treatment group, I doubt whether I can still use repeated ANOVA inculde confunding factors as covariates to interpret the connection between BDNF decrease and the course of treatment.

Thanks very much! I am looking forward to your help and answer.

I am sorry, the right expression is connection between BDNF increase and the course of treatment.

Hi Karen,

do you offer a discount to students? I did a study and wanted to ask some questions.

could’t find a contact email hence posting here.

Thanks

Hi

I have a similar question. If I do an ANCOVA, is there anyway to compare the change in each group? When I do an ANCOVA, I get a baseline for both groups, how do I compare this baseline score with the each group mean? Or can I?

Thank you so much

Hi Anoop,

You can’t. If change from baseline is what will answer your research question, don’t use ANCOVA.

Stumbled upon this post trying to address a reviewer comment that I should do an ANCOVA instead of a RM ANOVA. Thank you so much for your commentary, as this helped me in formulating my response.

Hi Jeff,

Woo hoo! That’s why I do this. π

I”m giving a pre post test questionnaire asking 10 questions with multiple answers but only one correct answer. Looking at knowledge deficit changes

in only one population. The comparator is the pre test score. A 1 hour educational module will be taken by each subject, then 6-8 weeks later they will take the same test again. Is ANCOVA the correct test to determine the difference? I really enjoy reading these online conversations. Thank you.

Hi Mimi,

It’s a possible approach, but there are many details that really affect the correct test. Start here:

http://www.theanalysisfactor.com/8-things-to-consider-in-choosing-statistical-analysis/

Hi Karen,

I conducted an intervention study where the groups were randomly assigned to experimental and control groups. The experimentals received treatment across two years and data were collected from experimentals and controls during 4 points in time. I want to evaluate the effectiveness of the intervention by comparing the changes across time between the two groups.

I initially thought I would need to do a repeated measures ANOVA but I read that the groups have to be the same. Does this mean that my control and experimental group are the same people? If I can’t use this analysis do you have another suggestion?

Thank you,

Debbie

Hi Debbie,

It sounds like you have a mixed model. One factor (time) is repeated measures but the other (group) is between subjects. So yes, you’ll need some version of a repeated measures, all of which can incorporate a between subjects factor.

Hi Karen, interesting post. Currently designing a study, confused whether I’m dealing with repeated measures analysis or not. Measuring X at 3 time points (different seasons) during the year – might be high in summer/low in winter, high in winter/low in summer, consistently high, consistently low, etc. Y (health outcome) measured at same 3 time points. Time between measures not equal between participants, since it’s the “season” that is of interest for X. Y may or may not change within the short time frame. Main research question is whether seasonal variation in X is associated with Y. ie. would someone with high-X in all seasons have healthier Y vs. someone with more variability in X, or someone with low-X in all 3 seasons? This almost seems to me a just a multiple linear/logistic regression analysis… but then I keep coming back and wondering about repeated measures or ANCOVA…? Thanks in advance for any insight.

Hi Katya,

If the three measurements are on the same individuals, then yes, it’s repeated measures. It doesn’t matter if time is equally spaced or if it really represents another concept, like season. You will definitely need a mixed model, not a repeated measures ANOVA. Mixed models can accommodate many more designs.

Hello,

I have a pre-test /post-test study design with two treatment and two control groups, all tested for attitude scores at 2 points of measurement. There are 2 additional demographic covariates that need to be controlled for. The study is not randomized and there were slight (albeit insignificant) differences in baseline scores for the 2 groups. Would an ANCOVA with attitude at t2 as DV and attitude at t1 as one of the 3 covariates be the right way to go here? I am interested in the effect of the intervention (i.e. whether the the intervention increases the attitude scores more than the expected increase with time).

Thanks in advance!

Emily

Hi Emily,

I would have said yes until I got to your last sentence. If the point is to test the INCREASE then the ANCOVA approach won’t do it. You’ll have to use a repeated measure approach.

I want to analyze effect on PSQI Score of 15 controls and 15 cases pre and post intervention (MSBR).What tests can I use to analyze ?

Hi SS,

I’d need to know how PSQI scores are measured. Are they categorical? Continuous, likert, counts?

Hi,

I have a stats related question as well…

I am testing for period effect in a crossover study that has multiple measure outcomes (i.e. test 1, test 2, test 3–but all these tests are within a similar theme, z-scored, and correlated (but not too highly)). I have run a one-way (or within group) repeated measure MANOVA but cannot quite resolve how to do the post-hoc paired t-tests…help pretty please!

Post hoc tests are tricky in a manova as you can really only do it on the univariate anovas. That may or may not come out the same as the multivariate tests.

great post – thanks a lot!

Thanks Karen.

Another question: how would I analyse my pre-test/ post-test data using the ANCOVA approach but with a hypothesis that the results are mediated by another variable.

For example, an online intervention program is effective in reducing the stigma toward mental health. However, I believe that this result is mediated by knowledge about depression.

Do I perform the same analysis but also add knowledge about depression as another covariate?

Hi Karen,

Can you please outline the assumptions I need to check for when using the ANCOVA approach you mention here?

Thanks

JCL

Hi JCL,

Yes, they’re listed here: http://www.theanalysisfactor.com/assumptions-of-linear-models/

Hi Karen, I’ve got data from a before-and-after study design with an intervention and control group. I’m want to see how my “intervention” (taking my obesity course) impacts weight-related attitudes compared to a “control” course. Attitudinal measures were done at the beginning (time 1) and end of the semester (time 2). However, there are signficant group differences in both attitudinal measures at baseline. What’s the best way to deal with this?

Option 1: Repeated measures ANOVA set up normally

Option 2: ANCOVA where anti-fat attitudes (time 2) is the DV and anti-fat attitudes (time 1) is a covariate

Option 3: Repeated measures ANOVA where the DIFFERENCE in anti-fat attitudes (time 2 – time 1) are the DV and the time 1 scores are a covariate

Or something completely different??

Hi Lori,

I think option 1 is best. Both 2 and 3 will allow you to look at what is happening at time 2, controlling for what is going on at time 1, but neither will allow you to show that it’s happening, and you do need to show that.

I’ve gain a lot with few minutes interacting with this content. I’ll very much appreciate if you’ll reply whenever I request for guide.

Andy

Best wishes

Hi Karen,

I have a situation as you described with Nancy’s example and I was wondering whether repeated measures ANOVA is appropriate to use.

I have a cohort with cardiac function assessment at two time points (t=0 and t=6 months) and blood pressure measurement at t=6.

The question that I like to answer is whether hypertension ( which is developped slowly over time but was diagnosed at t=6) is related to cardiac function depression at t=6.

For this purpose I would like to use RM ANOVA and enter cardiac function assessment as dependent with 2 levels and hypertension as “Between-subjects Factors” in SPSS.

Is this correct and to I have to model differently as a covariate?

Thank you very much!

Best regards,

Ulx

Hi Ulx,

If cardiac function *depression* is defined as how it differs from baseline, then indeed, you need a repeated measures, not ANCOVA.

Hi Karen,

Have a related question on ANCOVA

I had an experiment of 15 snakes exposed to 2 prey types. The snakes ate prey type one and then prey type two few minutes later. My covariate here is the relative prey mass (RPM; prey mass/snake mass). My measures are the time it took for the snake to capture prey one vs prey two. Is that feasible to perform ANCOVA considering that the same snake have eaten both prey types (lack of independence?). I will appreciate any hint. Does ancova works fine with unequal sample size?

Thanks,

Mil

Hi Mil,

You are correct–with two observations per snake, you need to account for non-independence. I would start here:

http://www.theanalysisfactor.com/repeated-measures-approaches/

http://www.theanalysisfactor.com/longitudinal-repeated-measures/

You can think of the two prey as repeated measures per snake.

I have a pre test – post test within subjects design. 30 questions on each test that fall into 3 different constructs. Did dependent t-tests on combined constructs and each question, plus Cohen’s d effect sizes. What other higher level analysis should I perform?

HI DK, that entirely depends on your research question. What do you want to know from these data?

Hi,

First, thanks a lot for the explanation. I have the same exactly the research design (pre- & post-test with experimental & control group) and faced the same problem. I received mixed advice and confused with the appropriate method -ANOVA or ANCOVA.

If my research objective is to evaluate the effect of an intervention program, which statistical analysis would you suggest? Both group has the same sample size, were randomly assigned and demographic characteristic for both groups are homogeneous. Please kindly help as I am very confused with too many conflicting advises. Thank you in advance.

Hi Emma,

As you can probably tell from the article, it’s hard to know what is exactly the right analysis. I’d have to ask you a lot of questions about your exact, specific research questions and design in order to really advise.

Karen

Hi I wonder if you could help me,

I am currently doing my dissertation on how mindfulness can be an effective way to reduce stress and have done a 2 way mixed anova for the analysis and written up that section. However I also have data from a follow up questionnaire asking participants how many times a week they did the training. So I need to find out if there is a relationship between time spent doing the training (3 or less days or more than 3 days) and amount stress score decreased from pre to post test.

Its not a straight forward correlation because I dont have numbers for the decrease I just have 2 scores pre and post. I have no idea what test to do on SPSS.