• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
The Analysis Factor

The Analysis Factor

Statistical Consulting, Resources, and Statistics Workshops for Researchers

  • Home
  • About
    • Our Programs
    • Our Team
    • Our Core Values
    • Our Privacy Policy
    • Employment
    • Guest Instructors
  • Membership
    • Statistically Speaking Membership Program
    • Login
  • Workshops
    • Online Workshops
    • Login
  • Consulting
    • Statistical Consulting Services
    • Login
  • Free Webinars
  • Contact
  • Login

Strategies for Choosing and Planning a Statistical Analysis

by Karen Grace-Martin 6 Comments

The first real data set I ever analyzed was from my senior honors thesis as an undergraduate psychology major. I had taken both intro stats and an ANOVA class, and I applied all my new skills with gusto, analyzing every which way.

It wasn’t too many years into graduate school that I realized that these data analyses were a bit haphazard and not at all well thought out. 20 years of data analysis experience later and I realized that’s just a symptom of being an inexperienced data analyst.

But even experienced data analysts can get off track, especially with large data sets with many variables. It’s just so easy to try one thing, then another, and pretty soon you’ve spent weeks getting nowhere.try different versions of models or get distracted by interesting, but irrelevant, relationships among variables.

The lesson? Make a plan.

Make a Plan

According to Frank Scarpaci, owner of Project Designworks, there is a

“1:10:100 Rule:

Every dollar spent on planning and preparation saves $10 on
project work or $100 on fixing problems after the project is done.”

I’m pretty sure that ratio holds for not just money, but time and frustration. I mean, you’d rather spend an hour now planning the analysis than two weeks redoing it after reviewers rip it to shreds, right?

The best time to plan the analysis is before collecting data.

This prevents those (all too common) situations where you realize you needed another variable or you should have measured something differently. Grant applications force you to do this, but every study would benefit.

How do you plan it?

I find a great outline for an analysis plan comes from an article by Daryl Bem about writing journal articles. The most helpful part for planning is the section, “Presenting the Findings”. This section outlines 7 steps for reporting each finding. For planning purposes, I condense these into three:

  1. State the conceptual hypothesis you are asking
  2. Restate this hypothesis in the terms of the variables that measure the concept
  3. List the statistical test or method that will answer this question

Simply repeat these three steps for all hypotheses the study is set up to answer. Start with the most general and important, and work down from there.

The Research Question is Central

You may have noticed that at the center is the conceptual hypothesis, or in looser terms, the research question. Everything you run should ultimately move you toward answering the research questions.

Write down your research questions and tape it to the wall near your computer.

There may be additional analyses that support the main one, and you may or may not be able to plan for them. But they should still serve the overall purpose of answering the research question.

For example, always plan on running univariate and bivariate descriptives and graphs to get a sense of your variables and their most basic relationships before you do much else.

Likewise, If you know you will need to run a factor analysis to create an index variable or deal with inevitable missing data, plan for those too.

Even the best plans, though, are guidelines. Surprises do come up (both good and bad), and you will probably have to adjust it as you go along. But don’t let that stop you from planning.

When you don’t know which tests answer the research question

“But wait a minute. I know the research question. I just don’t know know which statistics to use to answer them. What about those?” (I can hear you right now.)

The third step in planning is to choose the statistical test(s) to answer that research question. It’s impossible to list all the things to consider in choosing a statistical test, and there often isn’t just one option.

But here are some general guidelines. The statistical test must:

1. Answer the research question.

If your research question requires controlling for covariates, your test needs to have that ability. If the research question is about group differences, the test needs to be able to compare groups. This is why being specific is so important.

 

2. Take into account the design of the study.

Unless it was designed to accommodate other situations, most statistical tests assume simple random samples of independent measurements. If your sample is stratified or clustered; if measurements are repeated over time or space; or some other design issue led measurements to be beyond simple, the test needs to accommodate that.

 

3. Take into account the level of measurement and distribution of the independent and dependent variables.

This will ultimately affect which assumptions are and are not met. The exact same research question from the same design will use different statistical methods if the dependent variable is measured by a categorical variable than if it’s measured by a numerical variable.

 

4. Deal with any issues in the data.

This includes influential outliers, multicollinearity, truncation and censoring, small sample sizes, and missing data. Unlike the three issues above, you can’t always anticipate data issues, and you can’t always deal with them in the main analysis. You may have to use preliminary tests to deal with them first.

 

Sometimes these are very straightforward and the appropriate analysis is clear. More often it’s not.

Sometimes you don’t realize the data issues or the variable types you’re working with until you dig into the data a bit. So yes, make a plan. It will still help you keep on track. But it is not written in stone and following it to the letter will only decrease the quality of your analysis.

This is a great time to talk it over with your statistical advisor.

 

Tagged With: Censoring, data analysis plan, level of measurement, Missing Data, Multicollinearity, outliers, Research Question, small sample, statistical distributions, Study design, truncation

Related Posts

  • What to Do When You Can’t Run the Ideal Analysis 
  • When To Fight For Your Analysis and When To Jump Through Hoops
  • Statistical Consulting 101: 4 Questions you Need to Answer to Choose a Statistical Method
  • Eight Data Analysis Skills Every Analyst Needs

Reader Interactions

Comments

  1. OSIEMO says

    September 28, 2020 at 6:18 am

    May I get a PDF of the above article.

    Reply
  2. Luis Gonzalo Morales says

    September 26, 2020 at 1:24 pm

    Dear Karen,
    Thank you very much for your thoughtful and really helpful suggestions.
    There is a lot to learn from them, in particular for students of Ecology and Environmental Studies who are about to start their research projects. They are particularly helpful for those working outdoors, away from the controlled environment of lab experiments.

    Stay safe & healthy,

    Have a happy week end

    luis gonzalo

    Reply
  3. Jacob says

    May 12, 2019 at 12:19 pm

    Dear Karen
    I really appreciate your good work. it is really an eye opener. I would like to have information or more detail on data coding and cleaning.
    Thank you

    Reply
  4. Don says

    July 26, 2017 at 7:35 pm

    I pulled up this site and let me tell you. It has a been up and down as to where to find the best statistical information. Well, I guess I need not research further. I can blocked these concept styles and later apply these strategies in business meeting. Assistance isn’t always recognized vehemently. Thanks for your assistance.

    Reply
    • Karen Grace-Martin says

      January 15, 2019 at 11:15 am

      Thanks, Don. Glad you find it helpful.

      Reply
  5. Chris Olusola Ogedengbe says

    August 30, 2016 at 10:31 am

    I have really benefited from your writings. I stumbled into your website when I was searching for some details on sample estimate. I do not have any questions now but I really appreciate your works.

    Many thanks

    Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Please note that, due to the large number of comments submitted, any questions on problems related to a personal study/project will not be answered. We suggest joining Statistically Speaking, where you have access to a private forum and more resources 24/7.

Primary Sidebar

This Month’s Statistically Speaking Live Training

  • January Member Training: A Gentle Introduction To Random Slopes In Multilevel Models

Upcoming Workshops

  • Logistic Regression for Binary, Ordinal, and Multinomial Outcomes (May 2021)
  • Introduction to Generalized Linear Mixed Models (May 2021)

Read Our Book



Data Analysis with SPSS
(4th Edition)

by Stephen Sweet and
Karen Grace-Martin

Statistical Resources by Topic

  • Fundamental Statistics
  • Effect Size Statistics, Power, and Sample Size Calculations
  • Analysis of Variance and Covariance
  • Linear Regression
  • Complex Surveys & Sampling
  • Count Regression Models
  • Logistic Regression
  • Missing Data
  • Mixed and Multilevel Models
  • Principal Component Analysis and Factor Analysis
  • Structural Equation Modeling
  • Survival Analysis and Event History Analysis
  • Data Analysis Practice and Skills
  • R
  • SPSS
  • Stata

Copyright © 2008–2021 The Analysis Factor, LLC. All rights reserved.
877-272-8096   Contact Us

The Analysis Factor uses cookies to ensure that we give you the best experience of our website. If you continue we assume that you consent to receive cookies on all websites from The Analysis Factor.
Continue Privacy Policy
Privacy & Cookies Policy

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled

Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.

Non-necessary

Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.