Comparing mixed multiple regression models

When to use:

Model comparison serves to choose the set of independent variables (fixed effects) that describes the data best with as few parameters as possible. It works like comparing multiple regression models: we compare a full model with a reduced model (without some of the variables), or let the program select the model automatically (backward, forward, stepwise or subset selection).

In mixed models one additional rule applies: the random structure is shared by all compared models. Selection concerns the fixed effects only, and the random intercept, random slopes and covariance type set in the window remain in every model. A variable that has been given a random slope stays in every compared model as a fixed effect (one cannot remove the fixed effect of a variable while keeping its random slope). Whether the random effects themselves are needed is decided by the LR tests in the mixed multiple regression report.

Definitions of measures

Models are compared with the information criteria AIC, AICc and BIC (the smaller, the better) and, for nested models, with the likelihood ratio test (LR): the statistic \(2(\ell_{full}-\ell_{reduced})\) follows a \(\chi^2\) distribution with degrees of freedom equal to the number of removed variables. A non-significant result means that the removed variables carry no information and the simpler model should be chosen; a significant result – that the full model is significantly better.

Out-of-sample evaluation of models

The information criteria and the LR test are determined on the same data on which the models were fitted, which makes the assessment of the models too optimistic. As in comparing multiple regression models, the analysis window therefore makes it possible to evaluate the models on observations that did not take part in fitting them (the Validation option): \(k\)-fold cross-validation or a random test set, with the split determined by the Seed. For each model the report then gives RMSE, MAE and the predictive \(R^2\), and in the model selection block – the error of the null model and the best model by validation (the one with the smallest RMSE); validation error can also be chosen as the automatic selection criterion. In mixed models two additional rules apply:

• whole groups are split – all the observations of one group (e.g. one patient) go to the same part of the data. Observations of the same group are correlated with each other, so dividing them between the training part and the evaluated part would understate the error. The number of folds \(k\) is limited to the number of groups, and the share of the test set means the percentage of groups;
• the prediction for a new group is based on the fixed effects only – the random effect of a group that was not in the training set is unknown, so its expected value, i.e. zero, is assumed. The validation measures therefore tell how well the fixed effects of the model predict the observations of a new group (e.g. a new patient), and they correspond to the marginal, not the conditional, \(R^2\). RMSE also includes the variability between groups, so it is larger than the residual standard deviation of the model.

The null model here is the model with the intercept, the shared random structure and the variables that were given a random slope (they stay in every model); it is fitted and evaluated in the same way as the other models.

How to set up the analysis

Statistics \(\to\) Mixed models \(\to\) Mixed multiple regression – comparing models.

In the window we indicate the dependent variable \(Y\), the variables of the full model and of the reduced model (or the automatic selection criterion), as in comparing multiple regression models. Mixed-model options:

• Grouping variable (ID) – the column identifying the group; shared by all models,
• Random slopes (from full-model X) – variables whose effect differs between groups; such a variable stays in every compared model,
• Random effects covariance – independent or unstructured (correlated), shared by all models,
• Model coefficients (selection: ML) – REML or ML for the coefficient tables; the selection itself is always ML,
• Fixed effects degrees of freedom – Satterthwaite or Wald z test in the coefficient tables,
• Criterion – the automatic selection criterion: AIC, AICc, BIC, p (LR / F test) with a threshold, or validation error,
• Validation – none, k-fold CV, k = with the number of folds (from 2 to 100, 10 by default) or test set, % = with the share of the test set as a percentage of the groups (from 5 to 90, 30 by default); whole groups are split,
• Seed – the seed of the random split (12345 by default); the same seed means the same split.

Results and interpretation

The report contains a frame describing the shared random structure (grouping variable, number of groups, random components, covariance), the table Information criteria and LR tests with \(-2LL\) (ML), AIC, AICc, BIC, marginal and conditional \(R^2\) of each model and the LR test between successive models, and the coefficient tables of the compared models (conditional fixed effects, with the random-effect variances of each model). If a singular fit occurred for any model, it is marked in the table. When validation is switched on, the report additionally contains the Out-of-sample evaluation block (the method of splitting, the seed, the split unit, the error of the null model, the best model by validation), and for each model – RMSE, MAE and the predictive \(R^2\).