Comparing mixed logistic regression models

When to use:

Model comparison serves to choose the set of independent variables (fixed effects), as in comparing logistic regression models: we compare a full model with a reduced model, or use automatic selection (backward, forward, stepwise, subsets). The random structure is shared by all compared models: the random intercept, random slopes and covariance set in the window apply to every model, and a variable with a random slope stays in every model as a fixed effect. Selection concerns the fixed effects only; whether a random slope is needed is decided by the LR test in the mixed logistic regression report.

Definitions of measures

Models are compared with the information criteria AIC, AICc, BIC (the smaller, the better), pseudo-\(R^2\) and the marginal and conditional \(R^2\), and nested models – with the likelihood ratio test: \(2(\ell_{full}-\ell_{reduced})\) follows a \(\chi^2\) distribution with degrees of freedom equal to the number of removed variables. The likelihood is always approximated by the same method (Laplace or quadrature) in all models, so it is comparable.

Out-of-sample evaluation of models

The information criteria and the LR test are determined on the same data on which the models were fitted, which makes the assessment of the models too optimistic. As in comparing logistic regression models, the analysis window therefore makes it possible to evaluate the models on observations that did not take part in fitting them (the Validation option): \(k\)-fold cross-validation or a random test set, with the split determined by the Seed. For each model the report then gives the mean deviance (\(-2\ln L/n\); this is the validation error), \(AUC\) and the Brier score, and in the model selection block – the error of the null model and the best model by validation; validation error can also be chosen as the automatic selection criterion. In mixed models two additional rules apply:

• whole groups are split – all the observations of one group (e.g. one ward) go to the same part of the data. Observations of the same group are correlated with each other, so dividing them between the training part and the evaluated part would understate the error. The number of folds \(k\) is limited to the number of groups, and the share of the test set means the percentage of groups (the split is not stratified by the dependent variable here);
• the prediction for a new group is based on the fixed effects only – the random effect of a group that was not in the training set is unknown, so its expected value, i.e. zero, is assumed. The validation measures therefore tell how well the fixed effects of the model predict the event in a new group (e.g. in a new ward).

The null model here is the model with the intercept, the shared random structure and the variables that were given a random slope (they stay in every model); it is fitted and evaluated in the same way as the other models.

How to set up the analysis

Statistics \(\to\) Mixed models \(\to\) Mixed logistic regression – comparing models.

We indicate the dependent variable \(Y\) with the event value (Group (1)), the variables of the full and reduced models (or the selection criterion) and the mixed-model options: Grouping variable (ID), Random slopes (from full-model X), Random effects covariance and Likelihood approximation – all shared by the compared models. Below the list of selection methods there are the fields: Criterion (AIC, AICc, BIC, p (LR / F test) with a threshold, or validation error), Validation (none, k-fold CV, k = with the number of folds, or test set, % = with the share of the test set as a percentage of the groups) and Seed of the random split (the same seed means the same split).

Results and interpretation

The report contains a frame describing the shared random structure, the table Information criteria and LR tests with \(-2LL\), AIC, AICc, BIC, marginal and conditional \(R^2\) of each model and the LR test between successive models, and the coefficient tables of the models with conditional odds ratios and the random-effect variance of each model. When validation is switched on, the report additionally contains the Out-of-sample evaluation block (the method of splitting, the seed, the split unit, the error of the null model, the best model by validation), and for each model – the mean deviance, \(AUC\) and the Brier score.