# How Confident Are You About Confidence Intervals?

Any time you report estimates of parameters in a statistical analysis, it’s important to include their confidence intervals.

How confident are you that you can explain what they mean? Even those of us who have a solid understand of confidence intervals get tripped up by the wording.

### The Wording for Describing Confidence Intervals

Let’s look at an example.

The average person’s IQ is 100. A new miracle drug was tested on an experimental group. It was found to improve the average IQ 10 points, from 100 to 110. The 95 percent confidence interval of the experimental group’s mean was 105 to 115 points.

Which if any of the following are true:

1. If you conducted the same experiment 100 times, the mean for each sample would fall within the range of this confidence interval, 105 to 115, 95 times.

2. The lower confidence level for 5 of the samples would be less than 105.

3. If you conducted the experiment 100 times, 95 times the confidence interval would contain the population’s true mean.

4. 95% of the observations of the population fall within the 105 to 115 confidence interval.

5. There is a 95% probability that the 105 to 115 confidence interval contains the population’s true mean.

### A Visual Description Really Helps

Not sure? To help you visualize what a confidence interval is, let’s generate a random population of 10,000 observations. The population’s mean is 110 with a standard deviation of 25.

Next, we will randomly draw a sample of 100 observations from this population.  Below, you see the mean, standard deviation and confidence interval of the sample’s mean.

So far, so good. In reality, we seldom draw more than one sample from a population when conducting a study. To help us visualize how confidence intervals change as the sample changes we will randomly draw 99 more samples. This lets us graph each sample’s mean and confidence interval.

The horizontal red line at 110 is the population mean. The red dots represent each sample’s mean.

What can we observe from this graph?

1. The lower and upper confidence limits are seldom if ever the same.

2. Some confidence intervals have a narrower range than others.

Keep in mind that all samples came from the same population.

### Comparing the Wording the the Visual

Let’s look now at the multiple choices to the quiz, starting with the first choice.

Does the mean of each sample fall within any one sample’s confidence interval 95 out of 100 times?

Well, that would depend upon which sample we chose. If we chose a sample whose mean is near 110 that might be true. But ff our sample was similar to one from either edge of the graph it would not be true. The problem is, we never know where our sample is in relation to other possible samples.

Response 1 is incorrect.

With regards to response 2, we can see that the lower confidence level is below 105 for a substantial number of samples.

So response 2 is incorrect.

How about response 4, 95% of the population falls within the confidence interval? Well, it was given that the population mean was 110 and the standard deviation was 25. As a result, 95% of the observation of the population are between 60 and 160.

Therefore, response 4 cannot be true.

Response 3 is correct. Approximately 95 out of 100 of the confidence intervals contain the population mean. The graph shows 91 out of 100 contain the true mean. If a different seed had been used to draw the samples, the results could have been more than 95 out of 100 confidence intervals containing the true mean. But on average, 95 out of 100 confidence intervals will contain the true mean.

Response #5 was shown to be correct when this blog post was originally posted. One of our readers felt that it was incorrect and included a link to this article: https://link.springer.com/article/10.3758/s13423-015-0947-8 . We appreciate our readers’ input.

A very important point to remember, expect a sample’s 95% confidence interval to not contain the population mean 5% of the time.

Jeff Meyer is a statistical consultant with The Analysis Factor, a stats mentor for Statistically Speaking membership, and a workshop instructor. Read more about Jeff here.

Effect Size Statistics
Statistical software doesn't always give us the effect sizes we need. Learn some of the common effect size statistics and the ways to calculate them yourself.

1. Ying Wang says

Would 1 be correct? From the 100 means (denoted by the red points), it is about 95 of them are within 105 to 115.

2. Jerry says

Whew! Those five answers are so close, you really have to carefully dissect them.

And this doesn’t even consider the issue of “prediction intervals” . . .

3. Martina says

The previous replies are right: response 5 is not correct as 110 is for sure in the interval [105,115]. As those replies are from 2019, I do not understand why it is still incorrect…

4. Daniel says

I think response #5 is not correct, only #3 is correct. Response #5 relates to the “Fundamental Confidence Fallacy”, see fallacy 1 in this paper: