R² is such a lovely statistic, isn’t it? Unlike so many of the others, it makes sense–the percentage of variance in Y accounted for by a model.
I mean, you can actually understand that. So can your grandmother. And the clinical audience you’re writing the report for.
A big R² is always good and a small one is always bad, right?
I’ve seen a lot of people get upset about small R² values, or any small effect size, for that matter. I recently heard a comment that no regression model with an R² smaller than .7 should even be interpreted.
Now, there may be a context in which that rule makes sense, but as a general rule, no.
Just because effect size is small doesn’t mean it’s bad, unworthy of being interpreted, or useless. It’s just small. Even small effect sizes can have scientific or clinical significance. It depends on your field.
For example, in a dissertation I helped a client with many years ago, the research question was about whether religiosity predicts physical health. (If you’ve been in any of my workshops, you’ll recognize this example–it’s a great data set. The model used frequency of religious attendance as an indicator of religiosity, and included a few personal and demographic control variables, including gender, poverty status, and depression levels, and a few others.
The model R² was about .04, although the model was significant.
It’s easy to dismiss the model as being useless. You’re only explaining 4% of the variation? Why bother?
But think about this. If you think about all of the things that might affect someone’s health, do you really expect religious attendance to be a major contributor?
Even though I’m not a health researcher, I can think of quite a few variables that I would expect to be much better predictors of health. Things like age, disease history, stress levels, family history of disease, job conditions.
And putting all of them into the model would indeed give better predicted values. If the only point of the model was prediction, my client’s model would do a pretty bad job. (Perhaps the 70% comment came from someone who only runs prediction models).
But it wasn’t. The point was to see if there was a small, but reliable relationship. And there was.
Do small effect sizes require larger samples to find significance? Sure. But this data set had over 5000 people. Not a problem.
Many researchers turned to using effect sizes because evaluating effects using p-values alone can be misleading. But effect sizes can be misleading too if you don’t think about what they mean within the research context.
Sometimes being able to easily improve an outcome by 4% is clinically or scientifically important. Sometimes it’s not even close enough. Sometimes it depends on how much time, effort, or money would be required to get a 4% improvement.
As much as we’d all love to have straight answers to what’s big enough, that’s not the job of any statistic. You’ve got to think about it and interpret accordingly.