Mixed and Multilevel Models

Multilevel Models with Crossed Random Effects

Crossed random effects models are a little trickier than most mixed models, but they are quite common in many fields. Recognizing when you have one and knowing how to analyze the data when you do are important statistical skills.

The Nested Multilevel Design

The most straightforward use of Mixed Models is when observations are clustered or nested in some higher group.

It’s also so common that it often has its own name: multilevel model.

Examples include studies where patients share the same doctor, plants grow in the same field, or participants respond to multiple experimental conditions.

The observations at Level 1 (patient, plant, response) are clustered at Level 2 (doctor, field, or participant). This makes the responses from the same cluster correlated.

In these models, the Level 2 cluster is often not of interest. It’s what we call a “blocking factor.” Even so, we need to control for its effects.

If the researcher would like to generalize the results to all doctors, fields, or participants, these clustering variables are random factors.

The observations of the dependent variable are always measured at Level 1 (the patient, plant, or time point). Predictor variables (fixed effects) can be measured at either Level 1 or Level 2. For example, number of years of experience of a doctor would be at Level 2, but patient age would be measured at Level 1.

We assume the observations within cluster are are correlated, but the observations between clusters are independent.

A third level is possible as well. This would happen if each doctor sees all their patients at one of four hospitals or each field has only one of 5 species.

The Crossed Multilevel Design

In one kind of 2-level model, there is not one random factor at Level 2, but two crossed factors.

Each observation at Level 1 is nested in the combination of these two random factors. These models need to be specified correctly to capture the effects of both random factors at Level 2.

Here are the same examples with crossed random effects:

Example 1:

Every patient (Level 1) sees their Doctor (Random Effect at Level 2) at one of four Hospitals (Random Effect at Level 2) for a study comparing a new drug treatment for diabetes to an old one.

Each doctor sees patients at each of the hospitals. Patient responses vary across doctors and hospitals.

Because each Patient sees a single doctor at a single hospital, patients are nested in the combination of Doctor and Hospital.

The response is measured at Level 1–the patient. Predictors can occur at Level 1 (age, diet) or either Level 2 factor (years of practice by doctor, size of hospital).

Example 2:

An agricultural study is studying plants in 6 fields.

While there are many species of plants in each field, the researcher randomly chooses 5 species to study.

Each individual plant (Level 1) lies within one combination of species and field. But since every species is in every field, Species and Field are crossed at Level 2.

The response is measured at Level 1–the plant. Predictors can occur at Level 1 (height of plant) or either Level 2 factor (fertilizers applied to the field, whether the species is native or introduced).

Example 3:

In a psychological experiment, subjects rate statements that describe behaviors done by a fictional person, Bob.

On each trial, subjects rate whether or not they find Bob friendly. The response time of the rating is recorded.

Each subject sees the same 10 friendly and 10 unfriendly behaviors. The behaviors are not in themselves of interest to the experimenter,but are representative of all friendly and unfriendly behaviors that Bob could perform.

Because responses to the same behavior tend to be similar, it is necessary to control for their effects. Each trial of the experiment (Level 1) is nested both within Subject and Behavior, which are both random effects at Level 2.

Subject and Behavior are crossed at Level 2 since every Subject rates every Behavior. The response is measured at Level 1–the trial. Predictors can occur at Level 1 (a distractor occurs on some trials) or either Level 2 factor (Behavior is friendly or not, Subject is put into positive, neutral, or negative mood).

————————————————————————————————–

Luckily, standard mixed modeling procedures such as SAS Proc Mixed, SPSS Mixed, Stat’s xtmixed, or R’s lmer can all easily run a crossed random effects model. (R’s lme can’t do it).

Use care, however, because like most mixed models, specifying a crossed random effects model correctly can be tricky.

Confusing Statistical Terms #3: Level

by Karen Grace-Martin 2 Comments

Level is a term in statistics that is confusing because it has multiple meanings in different contexts (much like alpha and beta).

There are three different uses of the term Level in statistics that mean completely different things. What makes this especially confusing is that all three of them can be used in the exact same research context.

I’ll show you an example of that at the end.

So when you’re talking to someone who is learning statistics or who happens to be thinking of that term in a different context, this gets especially confusing.

Levels of Measurement

The most widespread of these is levels of measurement. Stanley Stevens came up with this taxonomy of assigning numerals to variables in the 1940s. You probably learned about them in your Intro Stats course: the nominal, ordinal, interval, and ratio levels.

Levels of measurement is really a measurement concept, not a statistical one. It refers to how much and the type of information a variable contains. Does it indicate an unordered category, a quantity with a zero point, etc?

It is important in statistics because it has a big impact on which statistics are appropriate for any given variable. For example, you would not do the same test of association between two nominal variables as you would between two interval variables.

Levels of a Factor

A related concept is a Factor. Although Factor itself has multiple meanings in statistics, here we are talking about a categorical predictor variable.

The typical use of Factor as a categorical predictor variable comes from experimental design. In experimental design, the predictor variables (also often called Independent Variables) are generally categorical and nominal. They represent different experimental conditions, like treatment and control traditions.

Each of these categorical conditions is called a level.

Here are a few examples:

In an agricultural study, a fertilizer treatment variable has three levels: Organic fertilizer (composted manure); High concentration of chemical fertilizer; low concentration of chemical fertilizer.

In a medical study, a drug treatment has four levels: Placebo; low dosage; medium dosage; high dosage.

In a linguistics study, a word frequency variable has two levels: high frequency words; low frequency words.

Although this use of level is very widespread, I try to avoid it personally. Instead I use the word “value” or “category” both of which are accurate, but without other meanings.

Level in Multilevel Models

A completely different use of the term is in the context of multilevel models. Multilevel models is a term for some mixed models. (The terms multilevel models and mixed models are often used interchangably, though mixed model is a bit more flexible).

Multilevel models have two or more sources of random variation. A two level model has two sources of random variation and can have predictors at each level.

A common example is a model from a design where the response variable of interest is measured on students. It’s hard though, to sample students directly or to randomly assign them to treatments, since there is a natural clustering of students within schools.

So the resource-efficient way to do this research is to sample students within schools.

Predictors can be measured at the student level (eg. gender, SES, age) or the school level (enrollment, % who go on to college). The dependent variable has variation from student to student (level 1) and from school to school (level 2).

We always count these levels from the bottom up. So if we have students clustered within classroom and classroom clustered within school and school clustered within district, we have:

Level 1: Students
Level 2: Classroom
Level 3: School
Level 4: District

So this use of the term level describes the design of the study, not the measurement of the variables or the categories of the factors.

Putting them together

So this is the truly unfortunate part. There are situations where all three definitions of level are relevant within the same statistical analysis context.

I find this unfortunate because I think using the same word to mean completely different things just confuses people. But here it is:

Picture that study in which students are clustered within school (a two-level design). Each school is assigned to use one of three math curricula (the independent variable, which happens to be categorical).

So, the variable “math curriculum” is a factor with 3 levels (ie, three categories). Because those three categories of “math curriculum” are unordered, “math curriculum” has a nominal level of measurement. And since “math curriculum” is assigned to each school, it is considered a level 2 variable in the two-level model.

See the rest of the Confusing Statistical Terms series.

Assessing the Fit of Regression Models

by Karen Grace-Martin 36 Comments

A well-fitting regression model results in predicted values close to the observed data values. The mean model, which uses the mean for every predicted value, generally would be used if there were no informative predictor variables. The fit of a proposed regression model should therefore be better than the fit of the mean model.

Three statistics are used in Ordinary Least Squares (OLS) regression to evaluate model fit: R-squared, the overall F-test, and the Root Mean Square Error (RMSE). All three are based on two sums of squares: Sum of Squares Total (SST) and Sum of Squares Error (SSE). SST measures how far the data are from the mean, and SSE measures how far the data are from the model’s predicted values. Different combinations of these two values provide different information about how the regression model compares to the mean model.

R-squared and Adjusted R-squared

The difference between SST and SSE is the improvement in prediction from the regression model, compared to the mean model. Dividing that difference by SST gives R-squared. It is the proportional improvement in prediction from the regression model, compared to the mean model. It indicates the goodness of fit of the model.

R-squared has the useful property that its scale is intuitive: it ranges from zero to one, with zero indicating that the proposed model does not improve prediction over the mean model, and one indicating perfect prediction. Improvement in the regression model results in proportional increases in R-squared.

One pitfall of R-squared is that it can only increase as predictors are added to the regression model. This increase is artificial when predictors are not actually improving the model’s fit. To remedy this, a related statistic, Adjusted R-squared, incorporates the model’s degrees of freedom. Adjusted R-squared will decrease as predictors are added if the increase in model fit does not make up for the loss of degrees of freedom. Likewise, it will increase as predictors are added if the increase in model fit is worthwhile. Adjusted R-squared should always be used with models with more than one predictor variable. It is interpreted as the proportion of total variance that is explained by the model.

There are situations in which a high R-squared is not necessary or relevant. When the interest is in the relationship between variables, not in prediction, the R-square is less important. An example is a study on how religiosity affects health outcomes. A good result is a reliable relationship between religiosity and health. No one would expect that religion explains a high percentage of the variation in health, as health is affected by many other factors. Even if the model accounts for other variables known to affect health, such as income and age, an R-squared in the range of 0.10 to 0.15 is reasonable.

The F-test

The F-test evaluates the null hypothesis that all regression coefficients are equal to zero versus the alternative that at least one is not. An equivalent null hypothesis is that R-squared equals zero. A significant F-test indicates that the observed R-squared is reliable and is not a spurious result of oddities in the data set. Thus the F-test determines whether the proposed relationship between the response variable and the set of predictors is statistically reliable and can be useful when the research objective is either prediction or explanation.

RMSE

The RMSE is the square root of the variance of the residuals. It indicates the absolute fit of the model to the data–how close the observed data points are to the model’s predicted values. Whereas R-squared is a relative measure of fit, RMSE is an absolute measure of fit. As the square root of a variance, RMSE can be interpreted as the standard deviation of the unexplained variance, and has the useful property of being in the same units as the response variable. Lower values of RMSE indicate better fit. RMSE is a good measure of how accurately the model predicts the response, and it is the most important criterion for fit if the main purpose of the model is prediction.

The best measure of model fit depends on the researcher’s objectives, and more than one are often useful. The statistics discussed above are applicable to regression models that use OLS estimation. Many types of regression models, however, such as mixed models, generalized linear models, and event history models, use maximum likelihood estimation. These statistics are not available for such models.

Mixed Up Mixed Models

by Karen Grace-Martin 1 Comment

A great article for specifying Mixed Models in SAS:

Mixed up Mixed Models
by Robert Harner & P.M. Simpson

Anyone doing mixed modeling in SAS should read this paper, originally presented at SUGI: SAS Users Group International conference. It compares the output from Proc Mixed and Proc GLM when specified different ways. There are some subtle distinctions in the meaning of the defaults in the Repeated and Random statements, and this paper does an excellent job of clarifying them.

Specifying Fixed and Random Factors in Mixed Models

by Karen Grace-Martin 10 Comments

One of the difficult decisions to make in mixed modeling is deciding which factors are fixed and which are random. Correctly specifying the fixed and random factors of the model is vital to obtain accurate analyses.

The definitions in many texts often do not help with decisions to specify factors as fixed or random, since textbook examples are often artificial and hard to apply.

Furthermore, the same factor can often be considered fixed or random, depending on the objective; this article outlines a different way to think about fixed and random factors.

Consider an experiment that examines beetle damage on cucumbers. The experiment is replicated at five farms and on four fields at each farm.

There are two varieties of cucumbers, and beetle damage is assessed on each of 50 plants at the end of the season. The researcher is interested in comparing differences in how much damage the two varieties sustain.

The experiment then has the following factors: VARIETY, FARM, and FIELD.

Fixed factors can be thought of in terms of differences.

The effect of a categorical fixed factor is defined by differences from the overall mean, and the effect of a continuous fixed factor (usually called a covariate) is defined by its slope–how the mean of the dependent variable differs with differing values of the factor.

The output for fixed factors provides estimates for mean-differences or slopes.

Conclusions regarding fixed factors are particular to the values of these factors. For example, if one variety of cucumber is found to suffer significantly less damage than the other, this says nothing about cucumber varieties that were not tested.

Random factors, on the other hand, are defined by a distribution and not by differences.

The values of a random factor are assumed to be chosen from a population with a normal distribution with a certain variance.

The output for a random factor is an estimate of this variance and not a set of differences from a mean. Conclusions regarding random factors should be expressed in terms of variance.

For example, we may find that the variance among fields makes up a certain percentage of the overall variance in beetle damage.

Situations that indicate fixed factors:

The factor is the primary treatment that the researcher wants to compare. In our example, VARIETY is definitely fixed as the researcher wants to compare the mean beetle damage on the two varieties.
The factor is a secondary control variable, and the researcher wants to control for differences in this factor. If these farms were specifically chosen for some feature they had, such as specific soil types or topographies that may affect beetle damage, and if the researcher would like to compare the farms as representatives of those soil types, then FARM should be fixed.
The factor has only two values. Even if everything else indicates that a factor should be random, if it has only two values, the variance cannot be calculated, and it should be fixed.

Situations that indicate random factors:

1. The researcher is interested in quantifying how much of the overall variation to attribute to this factor. If the researcher was interested in how much of the variation in beetle damage was attributable to the farm at which the damage took place, FARM would be random.
2. The researcher is not interested in knowing which means differ, but wants to account for the variation in this factor. If the farms were chosen at random, FARM should be random.
  This choice of the specific farms involved in the study is key. If you can rerun the study using different specific farms–different values of the Farm factor–and still be able to draw the same conclusions, then Farm should be random. However, if you had wanted to compare or control for these particular farms, then Farm would be fixed.
3. The researcher would like to generalize the conclusions about this factor to the whole population. There is nothing about comparing these specific fields that is of interest to the researcher. Rather, the researcher wants to generalize the results of this experiment to all fields, so FIELD is random.
4. Any interaction with a random factor is also random.

How the factors of a model are specified can have great influence on the results of the analysis and on the conclusions drawn.