• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
The Analysis Factor

The Analysis Factor

Statistical Consulting, Resources, and Statistics Workshops for Researchers

  • Home
  • About
    • Our Programs
    • Our Team
    • Our Core Values
    • Our Privacy Policy
    • Employment
    • Guest Instructors
  • Membership
    • Statistically Speaking Membership Program
    • Login
  • Workshops
    • Online Workshops
    • Login
  • Consulting
    • Statistical Consulting Services
    • Login
  • Free Webinars
  • Contact
  • Login

How Confident Are You About Confidence Intervals?

by Jeff Meyer 2 Comments

The results of any statistical analysis should include the confidence intervals for estimated parameters.

How confident are you that you can explain what they mean? Even those of us who have a solid understand of confidence intervals can get tripped up by the wording.

Let’s look at an example.

The average person’s IQ is 100. A new miracle drug was tested on an experimental group. It was found to improve the average IQ 10 points, from 100 to 110. The 95 percent confidence interval of the experimental group’s mean was 105 to 115 points.

Which if any of the following are true:

1. If you conducted the same experiment 100 times, the mean for each sample would fall within the range of this confidence interval, 105 to 115, 95 times.

2. The lower confidence level for 5 of the samples would be less than 105.

3. If you conducted the experiment 100 times, 95 times the confidence interval would contain the population’s true mean.

4. 95% of the observations of the population fall within the 105 to 115 confidence interval.

5. There is a 95% probability that the 105 to 115 confidence interval contains the population’s true mean.

Not sure? To help you visualize what a confidence interval is we will generate a random population of 10,000 observations. The populations mean is 110 with a standard deviation of 25.

From this population we will randomly draw a sample of 100 observations.  This is the mean, standard deviation and confidence interval of the sample’s mean.

We seldom draw more than one sample from a population when conducting a study. To help us visualize how confidence intervals change as the sample changes we will randomly draw 99 more samples. We will graph each sample’s mean and confidence interval.

The horizontal red line at 110 is the population mean. The red dots represent the sample’s mean.

What can we observe from this graph?

1. The lower and upper confidence limits are seldom if ever the same.

2. Some confidence intervals have a narrower range than others.

Keep in mind that all samples came from the same population.

Let’s look now at the multiple choices to the quiz, starting with the first choice.

Does the mean of each sample fall within any one sample’s confidence interval 95 out of 100 times?

That would depend upon which sample we chose. If we chose a sample whose mean is near 110 that might be true. If our sample was similar to one from either edge of the graph it would not be true. The problem is, we never know where our sample is in relation to other possible samples.

Response 1 is incorrect.

With regards to response 2, we can see that the lower confidence level is below 105 for a substantial number of samples.

Response 2 is incorrect.

How about response 4, 95% of the population falls within the confidence interval? It was given that the population mean was 110 and the standard deviation was 25. As a result, 95% of the observation of the population are between 60 and 160.

Response 4 cannot be true.

Response 3 is correct. Approximately 95 out of 100 of the confidence intervals contain the population mean. The graph shows 91 out of 100 contain the true mean. If a different seed had been used to draw the samples, the results could have been more than 95 out of 100 confidence intervals containing the true mean. But on average, 95 out of 100 confidence intervals will contain the true mean.

Response #5 is correct as well. 3 and 5 imply the same thing but are said differently. This is a common theme in statistics.

A very important point to remember, expect a sample’s 95% confidence interval to not contain the population mean 5% of the time.

Jeff Meyer is a statistical consultant with The Analysis Factor, a stats mentor for Statistically Speaking membership, and a workshop instructor. Read more about Jeff here.

Effect Size Statistics
Statistical software doesn't always give us the effect sizes we need. Learn some of the common effect size statistics and the ways to calculate them yourself.

Tagged With: confidence interval, estimate sample sizes, sample size

Related Posts

  • How Does the Distribution of a Population Impact the Confidence Interval?
  • How to Interpret the Width of a Confidence Interval
  • Member Training: Statistical Rules of Thumb: Essential Practices or Urban Myths?
  • The Effect Size: The Most Difficult Step in Calculating Sample Size Estimates

Reader Interactions

Comments

  1. Daniel says

    August 12, 2019 at 8:29 am

    I think response #5 is not correct, only #3 is correct. Response #5 relates to the “Fundamental Confidence Fallacy”, see fallacy 1 in this paper:
    https://link.springer.com/article/10.3758/s13423-015-0947-8

    One way to remember what CI do is to think of the as “compatible intervals” (see https://www.nature.com/articles/d41586-019-00857-9), i.e. “all the values between the interval’s limits are reasonably compatible with the data, given the statistical assumptions used to compute the interval”. And further, “because the interval gives the values most compatible with the data, given the assumptions, it doesn’t mean values outside it are incompatible; they are just less compatible”.

    Reply
    • Laszlo says

      September 4, 2019 at 7:58 am

      Daniel’s comment is correct, #5 is not correct (under the intuitive meaning of “95% probability”), and its meaning is not the same as #3.

      However, the rest of the text is written well, and I especially like the plot of the 100 confidence intervals, which clearly shows the meaning of a confidence interval: about 5 (in this case, 9) out of 100 of the CIs do not contain the true mean. Meaning it either does or doesn’t, but for most samples it does. As an experiment results in a single sample, one cannot talk about probabilities here.

      Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Please note that, due to the large number of comments submitted, any questions on problems related to a personal study/project will not be answered. We suggest joining Statistically Speaking, where you have access to a private forum and more resources 24/7.

Primary Sidebar

Free Webinars

Binary, Ordinal, and Multinomial Logistic Regression for Categorical Outcomes (Signup)

This Month’s Statistically Speaking Live Training

  • April Member Training: Statistical Contrasts

Upcoming Workshops

  • Logistic Regression for Binary, Ordinal, and Multinomial Outcomes (May 2021)
  • Introduction to Generalized Linear Mixed Models (May 2021)

Read Our Book



Data Analysis with SPSS
(4th Edition)

by Stephen Sweet and
Karen Grace-Martin

Statistical Resources by Topic

  • Fundamental Statistics
  • Effect Size Statistics, Power, and Sample Size Calculations
  • Analysis of Variance and Covariance
  • Linear Regression
  • Complex Surveys & Sampling
  • Count Regression Models
  • Logistic Regression
  • Missing Data
  • Mixed and Multilevel Models
  • Principal Component Analysis and Factor Analysis
  • Structural Equation Modeling
  • Survival Analysis and Event History Analysis
  • Data Analysis Practice and Skills
  • R
  • SPSS
  • Stata

Copyright © 2008–2021 The Analysis Factor, LLC. All rights reserved.
877-272-8096   Contact Us

The Analysis Factor uses cookies to ensure that we give you the best experience of our website. If you continue we assume that you consent to receive cookies on all websites from The Analysis Factor.
Continue Privacy Policy
Privacy & Cookies Policy

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled

Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.

Non-necessary

Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.

SAVE & ACCEPT