Updated Dec 18, 2020 to add more detail
In your statistics class, your professor made a big deal about unequal sample sizes in one-way Analysis of Variance (ANOVA) for two reasons.
1. Because she was making you calculate everything by hand. Sums of squares require a different formula* if sample sizes are unequal, but statistical software will automatically use the right formula. So we’re not too concerned. We’re definitely using software.
2. Nice properties in ANOVA such as the Grand Mean being the intercept in an effect-coded regression model don’t hold when data are unbalanced. Instead of the grand mean, you need to use a weighted mean. That’s not a big deal if you’re aware of it.
But there are a few real issues with unequal sample sizes in ANOVA. They don’t invalidate an analysis, but it’s important to be aware of them as you’re interpreting your output.
Two Practical Issues for Unequal Sample Sizes in One-Way ANOVA
1. Assumption Robustness with Unequal Samples
The main practical issue in one-way ANOVA is that unequal sample sizes affect the robustness of the equal variance assumption.
ANOVA is considered robust to moderate departures from this assumption. But that’s not true when the sample sizes are very different. According to Keppel (1993), there is no good rule of thumb for how unequal the sample sizes need to be for heterogeneity of variance to be a problem.
So if you have equal variances in your groups and unequal sample sizes, no problem. If you have unequal variances and equal sample sizes, no problem.
The only problem is if you have unequal variances and unequal sample sizes.
2. Power with Unequal samples
The statistical power of a hypothesis test that compares groups is highest when groups have equal sample sizes.
Power is based on the smallest sample size, so while it doesn’t hurt power to have more observations in the larger group, it doesn’t help either.
So if you have a specific number of individuals to randomly assign to groups, you’ll have the most power if you assign them equally.
If your grouping is a natural one, you’re not making decisions based on a total number of individuals. It’s very common to just happen to get a larger sample of one group compared to the others.
That doesn’t bias your test or give you incorrect results. It just means the power you have is based on the smaller sample.
So if you have 30 individuals with Treatment A and 40 individuals with Treatment B and 300 controls, that’s fine. It’s just that you could have stopped with 30 controls. The extra 270 didn’t help the power of this particular test.
Yes, this all holds true for independent samples t-tests
Independent samples t-tests are essentially a simplificiation of a one-way ANOVA for only two groups. In fact, if you run your t-test as an ANOVA, you’ll get the same p-value. And the between-groups F statistic will be the square of the t statistic you got in your t-test.
(Really, try it…. pretty cool, huh?)
This means they work the same way. Unbalanced t-tests have the same practical issues with unequal samples, but it doesn’t otherwise affect the validity or bias in the test.
Problems in Factorial ANOVA
Factorial ANOVA includes all those ANOVA models with more than one crossed factor. It generally involves one or more interaction terms.
Real issues with unequal sample sizes do occur in factorial ANOVA in one situation: when the sample sizes are confounded in the two (or more) factors. Let’s unpack this.
For example, in a two-way ANOVA, let’s say that your two independent variables (factors) are Age (young vs. old) and Marital Status (married vs. not).
Let’s say there are twice as many young people as old. So unequal sample sizes.
And say the younger group has a much larger percentage of singles than the older group. In other words, the two factors are not independent of each other. The effect of marital status cannot be distinguished from the effect of age.
So you may get a big mean difference between the marital statuses, but it’s really being driven by age.
What about Chi Square Tests?
(This article is about ANOVA (and t-tests), but I’ve updated to include Chi-Square tests after getting a lot of questions).
There are a number of different chi-square tests, but the two that can seem concerning in this context are the Chi-Square Test of Independence and The Chi-Square Test of Homogeneity. Both have two categorical variables. Both count the the frequencies of the combinations of these categories.
They calculate the test statistic the same way. Without getting into the math, it’s basically a comparison of the actual frequencies of the combinations with the frequencies you’d expect under the null hypothesis.
And luckily, unequal sample sizes do not affect the ability to calculate that chi-square test statistic. It’s pretty rare to have equal sample sizes, in fact. The expected values take the sample sizes into account. So no problems at all here.
That said, when there is a third variable involved, you can have an issue with Simpson’s Paradox. You may or may not have collected that third variable, so it’s worth thinking about whether there could be something else that is creating an association in a combination of two groups of that third variable that doesn’t exist in each group alone.
But that’s not really an issue with unequal sample sizes. That’s an issue of omitting an important variable from an analysis.
Evening,
I am running a study on the differences between distance learning, hybrid learning, and face to face. I have three different numbers for the groups Hybrid (N= 312); Distance (N= 1131), and Face to Face (N= 1007). The syllabi are the same for each group. I was running a Krustal-Wallis Test to determine any significance between each group. Then it occurred to me that I had different numbers for each group. Is this a problem for this study?
David
Dear Sir
I have two variables with unequal sample size. I has 46 and 2nd has only 13 samples. How I will determine the significant difference among them
hello everyone..
i have conducted a clinical trial in four groups of animals with different sample
size A(6) animals B(7) animals C(6) animals D(10) animals …i have checked effect of different medicine …observation was taken repeatedly untill the desired results obtained …each animal show different time of recovery ..so i have different sample size with unequal repeated observation …please suggest me a proper statistical analysis model.
Hi, I am running a factorial anova with 3 factors and 2 groups as IV
My groups are divided as 28 and 19 for one group
And 21 and 25 for the other group
DO you think this might be a problem for me when running the anova?
Thanks
Is it true that I can also run two-way ANOVA with unequal sample size in Minitab?
Hi Joanne,
Minitab will let you run it, but be careful about the inferences you’re making.
Hi Karen,
I am conducting a linear regression on Valence ratings (continuous response variable) with Condition (4-level factor) and Culture (x-level factor) as explanatory variable, using non-parametric bootstrapping because of non-normality and most importantly heteroskedasticity in my data. I am wondering which culture groups I can include in my analyses, given that the number of observations per culture group in each condition are very unequal, going from 8 to 217. Do you think that I should aim for a minimum of, say, 20 observations per level of Culture in each level of Condition? Is there a rule of thumb about the ratio of minimum:maximum number of observations per cell? Or of minimum number of observations in general?
Thank you!
Hi Karen,
Thanks for the website!
I have a bit of a different question related to this topic. I have a repeated measures design with 3 conditions. In condition A there are 150 observations for each participant, in condition B I have 20 observations for each participant and in C I have 150 observations for each participant. The total number of participants is 24 (i.e., each of the 24 subjects did all the three conditions in the experiment).
How is repeated measures ANOVA affected by this unequal numbers of observations in each condition? Would you happen to know where I can read more about this?
Thank you!
Hi Ayala,
I think this may help:
https://www.theanalysisfactor.com/linear-mixed-models-for-missing-data-in-pre-post-studies/
https://www.theanalysisfactor.com/repeated-measures-approaches/
I have a question.
What do you suggest I do to compare the means of two groups when group 1 has a sample size of 17 persons and group 2 has a sample size of 82 persons?
The sample variance values for the two groups are not that different.
The sample means don’t look all that different.
The only concern I have is that group 1 has n1 = 17 respondents and group 2 has n2 = 82 respondents.
Any help is greatly appreciated.
Kenneth Lewis
A t-test is fine there.
Dear Karen,
Thank you for this article, it is very interesting.
I have a question linked to this problematic.
When applying a post-hoc test comparing each group of the ANOVA with only one (say vehicle group versus all group doses of a treatment; with a Dunnett step-down post-hoc comparison), and you chose to higher the sample size of the vehicle at the cost of other groups’ sample size, are there known scenarios in which the power of the comparisons will be higher than in the balanced design? (without an alpha risk inflation?)
Thank you in advance
I have 3 different levels of English proficiency taking 6 different tests. However, all these different groups have different numbers of examinees. The first group has 490 participants, the second group has 1919 participants and the third group has 529 participants. Thus, I can say that I have unequal sample sizes for Mixed ANOVA.
When I do the analysis by using SPSS, it calculates the sum of squares and degrees of freedom by using the minimum sample size of the first group, which is 490. Is there a way to make SPSS analyze all the data of the unequal groups?
Hi, does it matter if groups are marginally unequal, say by n 1? Thanks.
Hello; Some variables in my data set are non-normal and my data also not independent, the data have also unequal sample or unbalanced data. Please suggest me what statistical test I should adopt.
Thanks
Hi! You talk about real issues with unequal sample sizes/variances in factorial anova, is this less of an issue when there is only one IV?
Yes, exactly. In a one-way model, it’s just not a big deal. Only when interactions come into it.
Its not a comment but a question. How can I compute for a sample size when I have 2 groups to choose from? I mean, see I have data for the population size of both male and female households in a particular site however, they are unequal. I need respondents from each group because I am having a comparative analysis.
Hi Karen, Great site!!
I was wondering where can I find the formulas for calculating 2-way Anova for non-balanced samples.
Moreover, I’ll be happy know the differences between 2 way Anova with and without replication (and formulas for both cases will be great).
Best wishes,
Yohay.
Very educative discussions. I am working on a research, which entails 2way unequal sample size. i am wondering if the SPSS version 20 can perform such task because thats what i have on my system.
thanks as i look forward to hearing from you.
Hi Karen
I need your help regarding my project . I divided the patients according to severity of disease in three groups , Group A=55, Group B=29 and Group c=30. I want to apply one-way anova but my data is not normally distributed and i need mean and standard deviation . Is it ok that if i will continue with one way anova. In my second project i have two group Group A=79 and Group B=35 and i want to apply independent t-test but again problem is that the data is not normally distributed. Please suggest me.
I will be really grateful to you
Dr.Mudassar Imran
For each population,the response variable that you want to measure is not normally distributed,then if the sample size is large enough then there is no need for normality because the 3 sample size and 3 sample standard deviation will be close to 3 population parameters which is required if null hypothesis is true.
hello,
great discussion!
I was wondering if the repeated-measures ANOVA using STATISTICA software is adjusting the sums of squares equation for unequal samples size like SPSS does?
thanks!
Hi Ioana,
I have no idea. I don’t use Statistica. Anyone else know?
i am doing a study on the prevalence and patterns of urinary tract infection amongst pregnant women attending a particular hospital in my country, comparing them to the non pregnant controls. i attained my study sample based on the prevalence. please how do i attain a formula to calculate the sample size now since i have been asked to stratify my pregnant patients into ist , 2nd and 3rd trimester?
what is the formula to attain a sample size for the comparism of means of unequal groups?
google it…u will find the formula
Hi Karen,
I have a question related to unequal sample sizes. I have a 2 (language background first language speaker (L1)/second language (L2) speaker) x3 (visual status: early blind/late blind/sighted) design. I investigate whether it is an advantage to have become blind as a child when it comes to second language acquisition.
In total I have N80: 40 L1 speakers and 40 L2 speakers (equal sizes), and each of these two groups have 11 early blind, 9 late blind and 20 sighted participants. Are these unequal sample sizes related to visual status a problem when using a 2×3 Anova? What do you suggest?
Many many thanks!
Helena
Hello Sir,
Sir in my research study, i had done work in three groups Group A( n=50), Group B( n=50) and Group C(n=25), i have used one way anova. is there any problem for selection of uneven sample size of Group C or it may affect statistical analysis. Please sir advice mee.
Thanking u
Hi, I’m doing study for me Bachelor of Science thesis too. Currently, I’m having problem with data analysis. My experiment design is 3×3 factorial design which consists of two independent variables (frying temperature and frying duration). However, for the duration factor is abit special which it has different duration). The setting is fried at 140C for 4, 5, 6 minutes while 160 and 180C fried for 1,2 and 3mins). Shall I use one way ANOVA or two way to analyse the effect on my sample?
thanks
Hi Karen,
I’m looking at the spatial variation of fish parasites for my Bachelor of Science thesis. I want to compare mean parasite abundance between male (n=71) and female (n=105) fish. I log transformed the parasite data and it has a normal distribution and equal variances, I was just wondering if I can use a One-way ANOVA to compare the mean abundance between sexes or would it be safer to just apply a non parametric Mann-Whitney U or Kruskal Wallis Test. Hope to hear from you.
Hi Karen,
My experiment model have two factors – temperature and different time points. I performed 2-Way GLM for the unequal sample size I have. However, it seems that there is no effect from the interaction of two factors and the temperature itself. My question is that will the result of comparison between two temp at different time points be valid if I perform them using one-way GLM after the no significant finding in the initial 2-way GLM?
Thank,
Hi Karen,
I hope you can help me. I’m trying to finish a paper for this term. I’ve just run two ANCOVAs. There were no problems with outliers, some problems with normality (skew and kurtosis <|3| although formal tests were significant), no problems with collinearity, correlation between covariates or homogeneity of slopes. Levene's test was significant for both ANCOVAS. The cell sizes and SDs are as follows:
ANCOVA with DV "A"
N=30, SD=1.31
N=78, SD=1.16
N=55, SD=.88
N=171, SD =1.21
ANCOVA with DV "B"
N=30, SD=0.91
N=78, SD=0.89
N=55, SD=.74
N=171, SD =.72
I realize that the smallest group has the highest variance in both cases. I hate to transform variables since it makes interpretation so complicated. What other options do I have?
Help!
-Kalyani
in my paper i am comparing the psychological well being of orphan and non-orphan children.sample size of n1=166,n2=333.is there a problem in computing independent t-test?
Hi,
I have data of 2 years 2004 and 2008 and I realized the sample size is not equal for both of the year how can i do data cleaning in stata in this case..
Dear Karen
Thanks for the information that you provided here. I have the same issue. I have a caregiver group of 96 and 42 control participants that I compare them on one variables. I checked for the variance and there were no significant differences in the variance. so I guess that refer to that. However, do you know any published book that I can cite?
thanks
Hi Karen,
I am working with a data set that has n~200, n~13, n~20. I would like to do an ANOVA but I am not sure how to approach this. What was the sample 20/200 you mentioned? Would a weighted mean account for these differences? The number of samples is also related to the number of interesting components for that group (not due to poor sampling).
Hi, I found your website very helpful and have a few of questions:
1) I have data of a entire population and am comparing the means of three groups. Do I still need to do significance testing since this isn’t really sampling?
2) The 3 groups have very different size (1200, 12000, 40000). I found the data not normal, so can I just use Kruskall-Wallis test?
3) I understand ANOVA is popular but I never found any data set that is normal. i.e. shapiro wilks test or kolmogrv test always have sig. <0 so I kept on using Kruskall walli test. Is that ok?
Many thanks!
Hi Karen,
I’m doing an analysis on mechanical properties with one factor. I have 3 groups, group 1 (n=5), group 2 (n=9) and group 3 (n=8). I have read the comment people asked and the replied you have given. So am I right to say that for one way ANOVA, is alright to analysis different sample size per group.
Yes.
Hi I was wondering what the full reference is for Keppel (1993). I’m interested in looking at that paper. Thanks
Hi Olivia,
It’s a book, not a paper. “Design and Analysis: A Researcher’s Handbook.”