• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
The Analysis Factor

The Analysis Factor

Statistical Consulting, Resources, and Statistics Workshops for Researchers

  • Home
  • About
    • Our Programs
    • Our Team
    • Our Core Values
    • Our Privacy Policy
    • Employment
    • Guest Instructors
  • Membership
    • Statistically Speaking Membership Program
    • Login
  • Workshops
    • Online Workshops
    • Login
  • Consulting
    • Statistical Consulting Services
    • Login
  • Free Webinars
  • Contact
  • Login

When Unequal Sample Sizes Are and Are NOT a Problem in ANOVA

by Karen Grace-Martin 219 Comments

Updated Dec 18, 2020 to add more detail

In your statistics class, your professor made a big deal about unequal sample sizes in one-way Analysis of Variance (ANOVA) for two reasons.

1. Because she was making you calculate everything by hand.  Sums of squares require a different formula* if sample sizes are unequal, but statistical software will automatically use the right formula. So we’re not too concerned. We’re definitely using software.

2. Nice properties in ANOVA such as the Grand Mean being the intercept in an effect-coded regression model don’t hold when data are unbalanced.  Instead of the grand mean, you need to use a weighted mean.  That’s not a big deal if you’re aware of it.

But there are a few real issues with unequal sample sizes in ANOVA. They don’t invalidate an analysis, but it’s important to be aware of them as you’re interpreting your output.

Two Practical Issues for Unequal Sample Sizes in One-Way ANOVA

1. Assumption Robustness with Unequal Samples

The main practical issue in one-way ANOVA is that unequal sample sizes affect the robustness of the equal variance assumption.

ANOVA is considered robust to moderate departures from this assumption. But that’s not true when the sample sizes are very different.  According to Keppel (1993), there is no good rule of thumb for how unequal the sample sizes need to be for heterogeneity of variance to be a problem.

So if you have equal variances in your groups and unequal sample sizes, no problem. If you have unequal variances and equal sample sizes, no problem.

The only problem is if you have unequal variances and unequal sample sizes.

2. Power with Unequal samples

The statistical power of a hypothesis test that compares groups is highest when groups have equal sample sizes.

Power is based on the smallest sample size, so while it doesn’t hurt power to have more observations in the larger group, it doesn’t help either.

So if you have a specific number of individuals to randomly assign to groups, you’ll have the most power if you assign them equally.

If your grouping is a natural one, you’re not making decisions based on a total number of individuals. It’s very common to just happen to get a larger sample of one group compared to the others.

That doesn’t bias your test or give you incorrect results. It just means the power you have is based on the smaller sample.

So if you have 30 individuals with Treatment A and 40 individuals with Treatment B and 300 controls, that’s fine. It’s just that you could have stopped with 30 controls. The extra 270 didn’t help the power of this particular test.

Yes, this all holds true for independent samples t-tests

Independent samples t-tests are essentially a simplificiation of a one-way ANOVA for only two groups. In fact, if you run your t-test as an ANOVA, you’ll get the same p-value. And the between-groups F statistic will be the square of the t statistic you got in your t-test.

(Really, try it…. pretty cool, huh?)

This means they work the same way. Unbalanced t-tests have the same practical issues with unequal samples, but it doesn’t otherwise affect the validity or bias in the test.

 

Problems in Factorial ANOVA

Factorial ANOVA includes all those ANOVA models with more than one crossed factor. It generally involves one or more interaction terms.

Real issues with unequal sample sizes do occur in factorial ANOVA in one situation: when the sample sizes are confounded in the two (or more) factors. Let’s unpack this.

For example, in a two-way ANOVA, let’s say that your two independent variables (factors) are Age (young vs. old) and Marital Status (married vs. not).

Let’s say there are twice as many young people as old. So unequal sample sizes.

And say the younger group has a much larger percentage of singles than the older group.  In other words, the two factors are not independent of each other.  The effect of marital status cannot be distinguished from the effect of age.

So you may get a big mean difference between the marital statuses, but it’s really being driven by age.

What about Chi Square Tests?

(This article is about ANOVA (and t-tests), but I’ve updated to include Chi-Square tests after getting a lot of questions).

There are a number of different chi-square tests, but the two that can seem concerning in this context are the Chi-Square Test of Independence and The Chi-Square Test of Homogeneity. Both have two categorical variables. Both count the the frequencies of the combinations of these categories.

They calculate the test statistic the same way. Without getting into the math, it’s basically a comparison of the actual frequencies of the combinations with the frequencies you’d expect under the null hypothesis.

And luckily, unequal sample sizes do not affect the ability to calculate that chi-square test statistic. It’s pretty rare to have equal sample sizes, in fact. The expected values take the sample sizes into account. So no problems at all here.

That said, when there is a third variable involved, you can have an issue with Simpson’s Paradox. You may or may not have collected that third variable, so it’s worth thinking about whether there could be something else that is creating an association in a combination of two groups of that third variable that doesn’t exist in each group alone.

But that’s not really an issue with unequal sample sizes. That’s an issue of omitting an important variable from an analysis.

Four Critical Steps in Building Linear Regression Models
While you’re worrying about which predictors to enter, you might be missing issues that have a big impact your analysis. This training will help you achieve more accurate results and a less-frustrating model building experience.

Tagged With: analysis of variance, ANOVA, SPSS, Unequal sample sizes

Related Posts

  • Same Statistical Models, Different (and Confusing) Output Terms
  • Why ANOVA and Linear Regression are the Same Analysis
  • 3 Reasons Psychology Researchers should Learn Regression
  • What are Sums of Squares?

Reader Interactions

Comments

  1. David M Scott says

    May 29, 2020 at 12:14 am

    Evening,
    I am running a study on the differences between distance learning, hybrid learning, and face to face. I have three different numbers for the groups Hybrid (N= 312); Distance (N= 1131), and Face to Face (N= 1007). The syllabi are the same for each group. I was running a Krustal-Wallis Test to determine any significance between each group. Then it occurred to me that I had different numbers for each group. Is this a problem for this study?

    David

    Reply
  2. MAK says

    November 12, 2019 at 4:29 am

    Dear Sir
    I have two variables with unequal sample size. I has 46 and 2nd has only 13 samples. How I will determine the significant difference among them

    Reply
  3. Dr. Qaiser Abbas says

    October 7, 2019 at 7:06 am

    hello everyone..
    i have conducted a clinical trial in four groups of animals with different sample
    size A(6) animals B(7) animals C(6) animals D(10) animals …i have checked effect of different medicine …observation was taken repeatedly untill the desired results obtained …each animal show different time of recovery ..so i have different sample size with unequal repeated observation …please suggest me a proper statistical analysis model.

    Reply
  4. franciele says

    August 12, 2019 at 6:47 am

    Hi, I am running a factorial anova with 3 factors and 2 groups as IV
    My groups are divided as 28 and 19 for one group
    And 21 and 25 for the other group
    DO you think this might be a problem for me when running the anova?

    Thanks

    Reply
  5. Joanne says

    January 25, 2019 at 9:37 am

    Is it true that I can also run two-way ANOVA with unequal sample size in Minitab?

    Reply
    • Karen Grace-Martin says

      March 4, 2019 at 11:18 am

      Hi Joanne,
      Minitab will let you run it, but be careful about the inferences you’re making.

      Reply
  6. Per says

    January 15, 2019 at 1:31 pm

    Hi Karen,

    I am conducting a linear regression on Valence ratings (continuous response variable) with Condition (4-level factor) and Culture (x-level factor) as explanatory variable, using non-parametric bootstrapping because of non-normality and most importantly heteroskedasticity in my data. I am wondering which culture groups I can include in my analyses, given that the number of observations per culture group in each condition are very unequal, going from 8 to 217. Do you think that I should aim for a minimum of, say, 20 observations per level of Culture in each level of Condition? Is there a rule of thumb about the ratio of minimum:maximum number of observations per cell? Or of minimum number of observations in general?
    Thank you!

    Reply
  7. Ayala says

    October 5, 2018 at 9:53 am

    Hi Karen,

    Thanks for the website!
    I have a bit of a different question related to this topic. I have a repeated measures design with 3 conditions. In condition A there are 150 observations for each participant, in condition B I have 20 observations for each participant and in C I have 150 observations for each participant. The total number of participants is 24 (i.e., each of the 24 subjects did all the three conditions in the experiment).

    How is repeated measures ANOVA affected by this unequal numbers of observations in each condition? Would you happen to know where I can read more about this?

    Thank you!

    Reply
    • Karen Grace-Martin says

      October 26, 2018 at 5:11 pm

      Hi Ayala,

      I think this may help:
      https://www.theanalysisfactor.com/linear-mixed-models-for-missing-data-in-pre-post-studies/
      https://www.theanalysisfactor.com/repeated-measures-approaches/

      Reply
  8. Kenneth Lewis says

    September 16, 2017 at 11:09 pm

    I have a question.
    What do you suggest I do to compare the means of two groups when group 1 has a sample size of 17 persons and group 2 has a sample size of 82 persons?
    The sample variance values for the two groups are not that different.
    The sample means don’t look all that different.
    The only concern I have is that group 1 has n1 = 17 respondents and group 2 has n2 = 82 respondents.
    Any help is greatly appreciated.

    Kenneth Lewis

    Reply
    • Karen says

      September 21, 2017 at 4:40 pm

      A t-test is fine there.

      Reply
  9. Yacine HAJJI says

    August 31, 2017 at 4:39 am

    Dear Karen,

    Thank you for this article, it is very interesting.
    I have a question linked to this problematic.

    When applying a post-hoc test comparing each group of the ANOVA with only one (say vehicle group versus all group doses of a treatment; with a Dunnett step-down post-hoc comparison), and you chose to higher the sample size of the vehicle at the cost of other groups’ sample size, are there known scenarios in which the power of the comparisons will be higher than in the balanced design? (without an alpha risk inflation?)

    Thank you in advance

    Reply
  10. Baran says

    December 4, 2016 at 3:35 am

    I have 3 different levels of English proficiency taking 6 different tests. However, all these different groups have different numbers of examinees. The first group has 490 participants, the second group has 1919 participants and the third group has 529 participants. Thus, I can say that I have unequal sample sizes for Mixed ANOVA.
    When I do the analysis by using SPSS, it calculates the sum of squares and degrees of freedom by using the minimum sample size of the first group, which is 490. Is there a way to make SPSS analyze all the data of the unequal groups?

    Reply
  11. Amber says

    September 28, 2016 at 7:16 am

    Hi, does it matter if groups are marginally unequal, say by n 1? Thanks.

    Reply
  12. Muhammad I Khan says

    September 9, 2016 at 12:45 pm

    Hello; Some variables in my data set are non-normal and my data also not independent, the data have also unequal sample or unbalanced data. Please suggest me what statistical test I should adopt.

    Thanks

    Reply
  13. A. S. says

    June 15, 2016 at 5:07 pm

    Hi! You talk about real issues with unequal sample sizes/variances in factorial anova, is this less of an issue when there is only one IV?

    Reply
    • Karen says

      June 17, 2016 at 9:53 am

      Yes, exactly. In a one-way model, it’s just not a big deal. Only when interactions come into it.

      Reply
  14. Tonette Aclan says

    December 8, 2015 at 7:39 pm

    Its not a comment but a question. How can I compute for a sample size when I have 2 groups to choose from? I mean, see I have data for the population size of both male and female households in a particular site however, they are unequal. I need respondents from each group because I am having a comparative analysis.

    Reply
  15. Yohay says

    September 8, 2015 at 7:55 am

    Hi Karen, Great site!!

    I was wondering where can I find the formulas for calculating 2-way Anova for non-balanced samples.
    Moreover, I’ll be happy know the differences between 2 way Anova with and without replication (and formulas for both cases will be great).

    Best wishes,
    Yohay.

    Reply
  16. Demtiw says

    May 19, 2015 at 12:29 am

    Very educative discussions. I am working on a research, which entails 2way unequal sample size. i am wondering if the SPSS version 20 can perform such task because thats what i have on my system.

    thanks as i look forward to hearing from you.

    Reply
  17. Imran says

    April 4, 2015 at 9:59 am

    Hi Karen

    I need your help regarding my project . I divided the patients according to severity of disease in three groups , Group A=55, Group B=29 and Group c=30. I want to apply one-way anova but my data is not normally distributed and i need mean and standard deviation . Is it ok that if i will continue with one way anova. In my second project i have two group Group A=79 and Group B=35 and i want to apply independent t-test but again problem is that the data is not normally distributed. Please suggest me.

    I will be really grateful to you

    Dr.Mudassar Imran

    Reply
    • ria sadhu says

      April 20, 2018 at 8:42 am

      For each population,the response variable that you want to measure is not normally distributed,then if the sample size is large enough then there is no need for normality because the 3 sample size and 3 sample standard deviation will be close to 3 population parameters which is required if null hypothesis is true.

      Reply
  18. ioana says

    March 3, 2015 at 2:57 pm

    hello,
    great discussion!
    I was wondering if the repeated-measures ANOVA using STATISTICA software is adjusting the sums of squares equation for unequal samples size like SPSS does?
    thanks!

    Reply
    • Karen says

      March 6, 2015 at 4:53 pm

      Hi Ioana,

      I have no idea. I don’t use Statistica. Anyone else know?

      Reply
  19. chinedu says

    November 27, 2014 at 5:58 am

    i am doing a study on the prevalence and patterns of urinary tract infection amongst pregnant women attending a particular hospital in my country, comparing them to the non pregnant controls. i attained my study sample based on the prevalence. please how do i attain a formula to calculate the sample size now since i have been asked to stratify my pregnant patients into ist , 2nd and 3rd trimester?

    Reply
  20. chinedu says

    November 27, 2014 at 5:40 am

    what is the formula to attain a sample size for the comparism of means of unequal groups?

    Reply
    • vipul says

      April 23, 2018 at 3:04 am

      google it…u will find the formula

      Reply
  21. Helena says

    August 14, 2014 at 3:15 am

    Hi Karen,

    I have a question related to unequal sample sizes. I have a 2 (language background first language speaker (L1)/second language (L2) speaker) x3 (visual status: early blind/late blind/sighted) design. I investigate whether it is an advantage to have become blind as a child when it comes to second language acquisition.

    In total I have N80: 40 L1 speakers and 40 L2 speakers (equal sizes), and each of these two groups have 11 early blind, 9 late blind and 20 sighted participants. Are these unequal sample sizes related to visual status a problem when using a 2×3 Anova? What do you suggest?

    Many many thanks!
    Helena

    Reply
  22. manisha says

    June 10, 2014 at 3:50 pm

    Hello Sir,
    Sir in my research study, i had done work in three groups Group A( n=50), Group B( n=50) and Group C(n=25), i have used one way anova. is there any problem for selection of uneven sample size of Group C or it may affect statistical analysis. Please sir advice mee.
    Thanking u

    Reply
  23. Mon says

    May 16, 2014 at 9:50 am

    Hi, I’m doing study for me Bachelor of Science thesis too. Currently, I’m having problem with data analysis. My experiment design is 3×3 factorial design which consists of two independent variables (frying temperature and frying duration). However, for the duration factor is abit special which it has different duration). The setting is fried at 140C for 4, 5, 6 minutes while 160 and 180C fried for 1,2 and 3mins). Shall I use one way ANOVA or two way to analyse the effect on my sample?

    thanks

    Reply
  24. Vicki says

    May 2, 2014 at 5:55 am

    Hi Karen,

    I’m looking at the spatial variation of fish parasites for my Bachelor of Science thesis. I want to compare mean parasite abundance between male (n=71) and female (n=105) fish. I log transformed the parasite data and it has a normal distribution and equal variances, I was just wondering if I can use a One-way ANOVA to compare the mean abundance between sexes or would it be safer to just apply a non parametric Mann-Whitney U or Kruskal Wallis Test. Hope to hear from you.

    Reply
  25. Yang says

    May 1, 2014 at 2:08 pm

    Hi Karen,

    My experiment model have two factors – temperature and different time points. I performed 2-Way GLM for the unequal sample size I have. However, it seems that there is no effect from the interaction of two factors and the temperature itself. My question is that will the result of comparison between two temp at different time points be valid if I perform them using one-way GLM after the no significant finding in the initial 2-way GLM?

    Thank,

    Reply
  26. Kalyani says

    April 30, 2014 at 2:53 pm

    Hi Karen,

    I hope you can help me. I’m trying to finish a paper for this term. I’ve just run two ANCOVAs. There were no problems with outliers, some problems with normality (skew and kurtosis <|3| although formal tests were significant), no problems with collinearity, correlation between covariates or homogeneity of slopes. Levene's test was significant for both ANCOVAS. The cell sizes and SDs are as follows:

    ANCOVA with DV "A"
    N=30, SD=1.31
    N=78, SD=1.16
    N=55, SD=.88
    N=171, SD =1.21

    ANCOVA with DV "B"
    N=30, SD=0.91
    N=78, SD=0.89
    N=55, SD=.74
    N=171, SD =.72

    I realize that the smallest group has the highest variance in both cases. I hate to transform variables since it makes interpretation so complicated. What other options do I have?

    Help!

    -Kalyani

    Reply
  27. Muluken Tigistu says

    April 24, 2014 at 12:13 pm

    in my paper i am comparing the psychological well being of orphan and non-orphan children.sample size of n1=166,n2=333.is there a problem in computing independent t-test?

    Reply
  28. trish says

    April 23, 2014 at 10:58 am

    Hi,

    I have data of 2 years 2004 and 2008 and I realized the sample size is not equal for both of the year how can i do data cleaning in stata in this case..

    Reply
  29. Somayyeh says

    April 22, 2014 at 6:44 am

    Dear Karen

    Thanks for the information that you provided here. I have the same issue. I have a caregiver group of 96 and 42 control participants that I compare them on one variables. I checked for the variance and there were no significant differences in the variance. so I guess that refer to that. However, do you know any published book that I can cite?

    thanks

    Reply
  30. TJ says

    April 13, 2014 at 5:46 pm

    Hi Karen,

    I am working with a data set that has n~200, n~13, n~20. I would like to do an ANOVA but I am not sure how to approach this. What was the sample 20/200 you mentioned? Would a weighted mean account for these differences? The number of samples is also related to the number of interesting components for that group (not due to poor sampling).

    Reply
  31. TW says

    April 11, 2014 at 10:00 am

    Hi, I found your website very helpful and have a few of questions:

    1) I have data of a entire population and am comparing the means of three groups. Do I still need to do significance testing since this isn’t really sampling?

    2) The 3 groups have very different size (1200, 12000, 40000). I found the data not normal, so can I just use Kruskall-Wallis test?

    3) I understand ANOVA is popular but I never found any data set that is normal. i.e. shapiro wilks test or kolmogrv test always have sig. <0 so I kept on using Kruskall walli test. Is that ok?

    Many thanks!

    Reply
  32. Josh says

    April 6, 2014 at 12:51 am

    Hi Karen,

    I’m doing an analysis on mechanical properties with one factor. I have 3 groups, group 1 (n=5), group 2 (n=9) and group 3 (n=8). I have read the comment people asked and the replied you have given. So am I right to say that for one way ANOVA, is alright to analysis different sample size per group.

    Reply
    • Karen says

      April 7, 2014 at 5:00 pm

      Yes.

      Reply
  33. Olivia says

    March 22, 2014 at 11:37 pm

    Hi I was wondering what the full reference is for Keppel (1993). I’m interested in looking at that paper. Thanks

    Reply
    • Karen says

      April 4, 2014 at 9:40 am

      Hi Olivia,

      It’s a book, not a paper. “Design and Analysis: A Researcher’s Handbook.”

      Reply
« Older Comments

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Please note that, due to the large number of comments submitted, any questions on problems related to a personal study/project will not be answered. We suggest joining Statistically Speaking, where you have access to a private forum and more resources 24/7.

Primary Sidebar

This Month’s Statistically Speaking Live Training

  • January Member Training: A Gentle Introduction To Random Slopes In Multilevel Models

Upcoming Workshops

  • Logistic Regression for Binary, Ordinal, and Multinomial Outcomes (May 2021)
  • Introduction to Generalized Linear Mixed Models (May 2021)

Read Our Book



Data Analysis with SPSS
(4th Edition)

by Stephen Sweet and
Karen Grace-Martin

Statistical Resources by Topic

  • Fundamental Statistics
  • Effect Size Statistics, Power, and Sample Size Calculations
  • Analysis of Variance and Covariance
  • Linear Regression
  • Complex Surveys & Sampling
  • Count Regression Models
  • Logistic Regression
  • Missing Data
  • Mixed and Multilevel Models
  • Principal Component Analysis and Factor Analysis
  • Structural Equation Modeling
  • Survival Analysis and Event History Analysis
  • Data Analysis Practice and Skills
  • R
  • SPSS
  • Stata

Copyright © 2008–2021 The Analysis Factor, LLC. All rights reserved.
877-272-8096   Contact Us

The Analysis Factor uses cookies to ensure that we give you the best experience of our website. If you continue we assume that you consent to receive cookies on all websites from The Analysis Factor.
Continue Privacy Policy
Privacy & Cookies Policy

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled

Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.

Non-necessary

Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.