There are 4 questions you must answer to choose an appropriate statistical analysis.
1. What is your Research Question?
2. What is the scale of measurement of the variables used to answer the research question?
3. What is the Design? (between subjects, within subjects, etc.)
4. Are there any data issues? (missing, censored, truncated, etc.)
But even once you’ve narrowed things down, you will nearly always still be left with options. For example, it may be that two different dependent variables both answer your research question well, but require different analyses because they are measured on different scales. Or you may get the same results by treating your censored data as missing.
I am not advocating data fishing or ignoring real issues like assumptions. But I am a pragmatist. The reality is that statistics is filled with grey areas, and there is often good justification for using a variety of statistical approaches. This article outlines 5 issues to consider in choosing an analysis where you have options. These are all issues I take into account with my statistical consulting clients.
Which approach to pursue depends on the answers to these questions:
1. What is the purpose of the research? Where will the results be published?
An unpublished honors or masters thesis has the primary goal of teaching the author, not obtaining scientific results (although that would be nice). If no one is ever going to read it, crude approaches are fine.
On the other extreme, if you’re publishing in Science, you’d better have every i dotted and every missing value multiply imputed even if you do get the same results from listwise deletion.
2. What resources do you have to conduct the analysis? What software do you have available? What is your deadline? How much time and money can you invest in learning or hiring someone to do an analysis for you?
Clearly, the fewer resources, the simpler the approach, taking into account the answers to the other questions.
3. What is your statistical background? What capacity to you have to correctly run and interpret the analysis?
Someone who is just beginning statistical analysis and has never done a regression will not be able to handle a multilevel model. End of story. If you don’t have the resources (question 2) to hire someone to do it, another approach will need to be considered. (And forget publishing in Science).
4.Will you ever have to do this analysis (or something similar) again?
Some masters and PHD students are not planning research careers. If this is the one and only statistical analysis you’ll ever do, it is less important for you to put the time and resources into mastering complicated statistical analysis.
On the other hand, journal editors are getting more sophisticated in the types of analyses they’re expecting. If you’re going to have to do it again and again, making the time to learn it now may well save much time and frustration over the years.
5. What are the expectations of the people who will be evaluating it–reviewers or committee?
This is especially important if the real purpose of this research is to finish your dissertation. If your stickler committee member wants it done a certain way, just do it. You can always redo your analyses once it’s time to publish.
Reviewers can be trickier. Decide how much you’re willing to fight. You may have to educate your reviewers (and editor) about a more appropriate technique that they don’t know about. Be prepared with lots of references and possibly include some educational text in the paper or cover letter. That said, I’ve witnessed researchers with PHDs in statistics jump through silly and unnecessary statistical hoops to please reviewers who thought they understood it better.