• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
The Analysis Factor

The Analysis Factor

Statistical Consulting, Resources, and Statistics Workshops for Researchers

  • Home
  • Our Programs
    • Membership
    • Online Workshops
    • Free Webinars
    • Consulting Services
  • About
    • Our Team
    • Our Core Values
    • Our Privacy Policy
    • Employment
    • Collaborate with Us
  • Statistical Resources
  • Contact
  • Blog
  • Login

Should Confidence Intervals or Tests of Significance be Used?

by Christos Giannoulis Leave a Comment

What is a Confidence Interval?

Any sample-based findings used to generalize a population are subject to sampling error. In other words, sample statistics won’t exactly match the population parameters they estimate.

One way of dealing with sampling error is to ignore results if there is a chance that they could be due to sampling error. This is the approach adopted with significance tests. Sample effects are treated as being zero if there is more than a 5 percent or 1 percent chance they were produced by sampling error.

Confidence intervals provide a useful alternative to significance tests. Instead of deciding whether the sample data support the devil’s argument that the null hypothesis is true we can take a less cut and dried approach.

We can take a range of values of a sample statistic that is likely to contain a population parameter. A certain percentage (confidence level) of intervals will include the population parameter in the long run (over repeated sampling).

For any given sample size, the wider the confidence interval, the higher the confidence level. If you want a more precise (i.e. narrower) confidence interval, you will have to use a lower level of confidence or use a larger sample.

We might find in a sample that 52 percent of respondents say they intend to vote for Party X at the next election. This figure is the sample estimate.

Since this came from a sample that inevitably has sampling error, we must allow a margin of error. Using the confidence interval, we can estimate the interval within which the population parameter is likely to lie.

This approach avoids the confusing logic of null hypothesis testing and its simplistic significant/not significant dichotomy.

Confidence intervals are a form of inferential analysis and can be used with many descriptive statistics such as percentages, percentage differences between groups, correlation coefficients and regression coefficients.

Like tests of significance, confidence intervals assume that the sample estimates come from a simple random sample. It is inappropriate to use these statistics on data from non-probability samples.

Novice researchers might find themselves in tempting situations to say that they are 95% confident that the confidence interval contains the true value of the population parameter. This is usually not technically correct (at least in frequentist statistics). Rather it is correct to say: Were one to take an infinite number of samples of the same size, on average 95% of them would produce confidence intervals containing the true population value.

Reasons to Use Confidence Intervals Over Significance Tests

Although tests of significance are used more than confidence intervals, many researchers prefer confidence intervals over tests of significance. These reasons include:

1. Confidence intervals provide all the information that a test of statistical significance provides and more.

If, at the 95 percent confidence level, a confidence interval for an effect includes 0 then the test of significance would also indicate that the sample estimate was not significantly different from 0 at the 5 percent level.

2. The confidence interval provides a sense of the size of any effect. The figures in a confidence interval are expressed in the descriptive statistic to which they apply (percentage, correlation, regression, etc.).

This effect size information is missing when a test of significance is used on its own. In our income example the interval estimate for the difference between male and female average incomes was between $2509 and $8088. This gives a sense of roughly what the actual difference is and also of the margin of error of any such difference.

3. Since confidence intervals avoid the term ‘significance’, they avoid the misleading interpretation of that word as ‘important.’

Confidence intervals remind us that any estimates are subject to error and that we can provide no estimate with absolute precision.

The Pathway: Steps for Staying Out of the Weeds in Any Data Analysis
Get the road map for your data analysis before you begin. Learn how to make any statistical modeling – ANOVA, Linear Regression, Poisson Regression, Multilevel Model – straightforward and more efficient.

Tagged With: confidence interval, p-value, sampling error, significance testing, statistical significance

Related Posts

  • Member Training: Inference and p-values and Statistical Significance, Oh My!
  • Member Training: Writing Up Statistical Results: Basic Concepts and Best Practices
  • Best Practices for Organizing your Data Analysis
  • Three Habits in Data Analysis That Feel Efficient, Yet are Not

Reader Interactions

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Please note that, due to the large number of comments submitted, any questions on problems related to a personal study/project will not be answered. We suggest joining Statistically Speaking, where you have access to a private forum and more resources 24/7.

Primary Sidebar

This Month’s Statistically Speaking Live Training

  • Member Training: Analyzing Pre-Post Data

Upcoming Free Webinars

Poisson and Negative Binomial Regression Models for Count Data

Upcoming Workshops

  • Analyzing Count Data: Poisson, Negative Binomial, and Other Essential Models (Jul 2022)
  • Introduction to Generalized Linear Mixed Models (Jul 2022)

Copyright © 2008–2022 The Analysis Factor, LLC. All rights reserved.
877-272-8096   Contact Us

The Analysis Factor uses cookies to ensure that we give you the best experience of our website. If you continue we assume that you consent to receive cookies on all websites from The Analysis Factor.
Continue Privacy Policy
Privacy & Cookies Policy

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
Non-necessary
Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.
SAVE & ACCEPT