One question I always get in my Repeated Measures Workshop is:
“Okay, now that I understand how to run a linear mixed model for my study, how do I write up the results?”
This is a great question.
There are many pieces of the linear mixed models output that are identical to those of any linear model–regression coefficients, F tests, means.
But there is also a lot that is new, like intraclass correlations and information criteria. And a lot of output we’re used to seeing, like R squared, isn’t there anymore.
And because the model is more complicated, you may need to include in your paper more information about how you set up the model. For example, you usually need to say whether you included a random intercept or slope (and at which level) and which covariance structure you chose for the residuals.
The problem I have in answering this is how you write it up is very much dependent on who you’re writing for.
Writing for Journals and Committees
The first thing to consider is your field and how familiar readers from your field will be with mixed models. I’ve worked with clients whose reviewers had never heard of them.
If you’re in a field like this, one of two things will happen.
1. reviewers will be suspicious that you were making up some hocus-pocus statistics to get significant p-values. Or
2. reviewers will have no idea what you’re talking about and can’t evaluate what you’ve done. Furthermore, they’ll insist you report statistics that aren’t available in mixed models, like eta-squared.
If that’s your situation, you’re going to have to write it up with a bit more detail than you otherwise would. Confused reviewers won’t be inclined to accept your paper. Educate your readers about the methods. Explain not just what you did, but why it was necessary.
Be generous with citations of not only papers that used mixed models but also those that explain what they are.
If you’re in a field where mixed models are more familiar and most readers will understand them, you’ll need to give enough detail that someone who understands mixed models could evaluate the approach. This means you will need to say which random effects you included and which covariance structure you chose. But you won’t have to explain what a random effect does.
Writing for non-statistical audiences
What if your audience isn’t a research audience, but your company’s marketing managers or your agency’s clinical staff? They likely never needed statistics classes and have no understanding at all of what you’re doing. They just want to understand whether the intervention worked and they’re counting on you to know what you’re doing statistically.
In that case, you want to eliminate as much statistical jargon as possible. Do everything you can to explain anything you can in English. You can always put the statistical details in an appendix in case some future researcher comes across it.
They may understand “I used a linear mixed model because it accounts for the fact that multiple responses from the same person are more similar than responses from other people.” But they won’t want to know how or why this is true.
Use a model
The ideal situation is to use as a guide a published paper that used the same type of mixed model in the journal you’re submitting to.
I can’t usually supply that to researchers, because I work with so many in different fields.
So I thought I’d try this. Here is a list of a few papers I’ve worked on personally that used mixed models.
Feel free to look them up, in case it helps.
- J Tee Todd, Susan G Butler, Drew P Plonk, Karen Grace-Martin, Cathy A Pelletier. (2012). Effects of chemesthetic stimuli mixtures with barium on swallowing apnea duration. The Laryngoscope, 122(10):2248-51.
- Catriona M Steele, Gemma L Bailey, Sonja M Molfenter, Erin M Yeates, Karen Grace-Martin. (2010). Pressure profile similarities between tongue resistance training tasks and liquid swallows. The Journal of Rehabilitation Research and Development, 47(7):651-60.
- Susan Butler, Karen Grace-Martin. (2010). The Effect of Chemesthesis on Swallowing Apnea Duration. Otolaryngology-Head and Neck Surgery, 143(2).
One other suggestion I’ve found helpful. Try googling:
Type that in exactly, with the quotes, but replace the word field with whatever your field is: nursing, sociology, etc. You will be surprised what you may find.
My request to you
If you have worked on or know of a paper that used mixed models, please give us the reference in the comments. Links to online versions are great too, if you have one.
Trust me, many people in your field are looking for an example and will be happy to cite it.
For those working in the area of clinical trials where Mixed Models for Repeated Measures (MMRM) is used fairly frequently for repeated measures (longitudinal) data then you can see many examples by doing a web search for the FDA Statistical Reviews of new drug applications. For example, in Google try ‘site:fda.gov “mixed models” ‘ (remove the single quotes) and you’ll see a number of PDFs containing the stat reviews involving mixed models.
a psych example:
Improvement in hypersomnia with high frequency repetitive transcranial magnetic stimulation in depressed adolescents: Preliminary evidence from an open-label study
A Irem Sonmez 1, M Utku Kucuker 1, Charles P Lewis 1, Bhanu Prakash Kolla 2, Deniz Doruk Camsari 1, Jennifer L Vande Voort 1, Kathryn M Schak 1, Simon Kung 1, Paul E Croarkin 3
https://pubmed.ncbi.nlm.nih.gov/31634515/
Found this example in tumor immunology (just an example, writeup is minimal): https://www.cell.com/immunity/fulltext/S1074-7613(18)30261-9
Here’s an example of a mixed model in an applied psychology journal
Kim, Block, & Nguyen (2019). What’s visible is my race, what’s invisible is my contribution: Understanding the effects of race and color-blind racial attitudes on the perceived impact of microaggressions toward Asians in the workplace. Journal of Vocational Behavior. 113, 75-87. https://doi.org/10.1016/j.jvb.2018.08.011
Thank you so much!!
Here are two papers in linguistics
Lukyanenko, C., & Fisher, C. (2016). Where are the cookies? Two-and three-year-olds use number-marked verbs to anticipate upcoming nouns. Cognition, 146, 349-370.
Kwon, H. (2017). Language experience, speech perception and loanword adaptation: Variable adaptation of English word-final plosives into Korean. Journal of Phonetics, 60, 1-19.
Thank you Sarah for posting these linguistics examples!
Found a marine biology and ecology paper that has a useful way of reporting mixed models. https://www.nature.com/articles/srep28875#t2
thank you Candace
Thanks so much for this, Karen. You made the work of researching how to report this specific for my audience seems much less daunting!
Here is a behavioral ecology paper that reports LMM
http://homepages.abdn.ac.uk/julienmartin/pages/content/uploads/Martin-Réale-2008-AB.pdf
Forgot to include a link to helpful tutorials I found (which include a little bit about how to report LMM) by Bodo Winter:
http://www.bodowinter.com/tutorials.html
Thank you very much for this. I have just been introduced to R to analyse my data for my final year BSc research project so this is very helpful. Thank you.
Thank you for the clear explanation, and for links to related topics. And, I appreciate the citations listed by other readers.
Another example Relationship Between Drug Dreams, Affect, and Craving
During Treatment for Substance Dependence
Hel´ ene Tanguay, MSc, Antonio Zadra, PhD, Daniel Good, BA, and Francesco Leri, PhD
These authors now explain mixed models to social psychologists:
Judd, C. M., Westfall, J., & Kenny, D. A. (2012). Treating stimuli as a random factor in social psychology: A new and comprehensive solution to a pervasive but largely ignored problem. Journal of personality and social psychology, 103(1), 54-69.
For the ecology field, the following paper uses linear mixed models:
XU, C., LETCHER, B. H. and NISLOW, K. H. (2010), Context-specific influence of water temperature on brook trout growth rates in the field. Freshwater Biology, 55: 2253–2264. doi:10.1111/j.1365-2427.2010.02430.x
Saviour!
Here is another one, from one of my clients:
http://www.tandfonline.com/doi/full/10.1080/1754730X.2015.1110495
Here’s a couple of articles using linear mixed models:
Schiefer, J. and Fischer C. (2008). The gap between wine expert ratings and consumer preferences: Measures, determinants and marketing implications. International Journal of Wine Business Research, Vol. 20, No. 4, 335 – 351.
Tainsky, S. (2009). “Television broadcast demand for National Football League contests.” Journal of Sports Economics 11(6): 629-640.
Sulmont-Rossé, C., et al. (2008). “Impact of the arousal potential of uncommon drinks on the repeated exposure effect.” Food Quality and Preference 19(4): 412-420.
Here are a couple of examples of mixed models used in articles in an APA journal, Emotion:
Denny, B. T., & Ochsner, K. N. (2014). Behavioral effects of longitudinal training in cognitive reappraisal. Emotion, 14(2), 425–33. doi:10.1037/a0035276
Crane, C. A., & Testa, M. (2014). Daily Associations Among Anger Experience and Intimate Partner Aggression Within Aggressive and Nonaggressive Community Couples. Emotion, 14(5), 985–994.