Comparing logistic regression models

When to use:

Because it is possible to analyse many independent variables simultaneously in a single logistic regression model, just as in multiple linear regression, the problem of choosing the optimal model arises. When selecting independent variables, you should keep in mind that the model should contain variables strongly correlated with the dependent variable and weakly correlated with one another.

Comparing logistic regression models lets you answer the question of whether adding (or removing) independent variables significantly changes the quality of the model. If two models do not differ significantly, it is worth choosing the simpler one – the one with fewer variables. In addition, when we are interested in the practical usefulness of a new risk factor, we can compare how much the model's ability to predict the event under study improves once this factor is added.

Assumptions

Definitions of measures

When comparing models with a different number of independent variables, we pay attention to the model fit and to the information criteria. We decide which model to choose on the basis of the values of \(R_{Pseudo}^2\), \(R^2_{Nagelkerke}\), \(R^2_{Cox-Snell}\) and the result of the likelihood ratio test, which compares successive (neighbouring) models as they are created. For every model the maximum of the likelihood function is calculated, and the likelihood ratio test checks whether the full and reduced models differ significantly in this respect. If the compared models do not differ significantly, we should choose the one with fewer variables – the lack of a difference means that the variables present in the full model but absent from the reduced model do not carry significant information. If, on the other hand, the difference is statistically significant, this means that one of the models (the one with more variables, a larger \(R^2\) and a lower value of the AIC, AICc or BIC information criterion) is significantly better than the other.

Comparing the predictive value of models – the regression models that are built let us predict the probability of the event under study occurring, based on the independent variables analysed. When many factors that increase the risk of the event are already known, an important criterion for a new candidate risk factor is how much it improves prediction performance once it is added to the model. For example, when studying risk factors for coronary heart disease (age, systolic and diastolic blood pressure, obesity, cholesterol, smoking), we can check whether adding a further factor significantly improves the estimation of disease risk. The predictive value of a model follows from the predicted probability determined for every person studied – the closer this value is to 1, the more likely the event is to occur. Based on the predicted probability, we can determine and compare between models the area \(AUC\) under the ROC curve, as well as the \(NRI\) and \(IDI\) coefficients.

• Change in the area under the ROC curve – in logistic regression models the ROC curve is built based on the classification of cases into the group experiencing the event or not, and on the predicted probability of the dependent variable. The larger the area under the curve, the more accurately the probability determined by the model predicts the actual occurrence of the event. By comparing the area under the ROC curve for the full and the reduced model, we therefore check whether adding the factors significantly improved the model's prediction (DeLong's method, described in more detail in the Comparing ROC curves chapter).

• Net Reclassification Improvement (\(NRI\)) – a measure based on the reclassification table, describing the upward or downward shift of the predicted probability once a new factor is added to the model. It is determined separately for objects experiencing the event and for those that did not experience it. \(NRI\) can be determined for a given split of the predicted probability into categories (categorical \(NRI\)) or without the need to determine categories (continuous \(NRI\)).

  • Categorical \(NRI\) requires the split points of the predicted probability to be set arbitrarily (a maximum of 9 points, i.e. a maximum of 10 categories; most often one or two points are used). The values of the categorical \(NRI\) can only be compared with one another when they are based on the same split points. For example, for the split points 0.1 and 0.3: if, in the "old" (smaller) model, a person obtained a probability below 0.1, and in the "new" model – between 0.1 and 0.3, then that person was reclassified upward. If the value from both models falls within the same interval, the person was not reclassified, and if the probability from the "new" model is lower than in the "old" model, we are dealing with a downward reclassification.

    regression models	reclassification upward		no reclassification		reclassification downward
    predicted probability	"old"	"new"	"old"	"new"	"old"	"new"
    [0.3 to 1]					\(\oplus\)
    [0.1; 0.3)		\(\oplus\)	\(\oplus\)	\(\oplus\)		\(\oplus\)
    [0; 0.1)	\(\oplus\)

  • Continuous \(NRI\) does not require categories to be determined arbitrarily – every change in probability, even the smallest, upward or downward relative to the probability from the "old" model, is treated as a move into the next category.
  Note! Using continuous \(NRI\) does not require the arbitrary definition of probability split points, but even small changes in risk (with no counterpart in clinical observation) can influence the value of this coefficient. Categorical \(NRI\), on the other hand, reflects only those changes that the researcher judged to be important when defining the probability split points.

  The overall \(NRI\) coefficient lies in the range from \(-2\) to \(2\), while the partial coefficients \(NRI_{events}\) and \(NRI_{nonevents}\) lie in the range from \(-1\) to \(1\) (from \(-100\%\) to \(100\%\)). Positive values indicate a favourable reclassification, and negative values – an unfavourable reclassification, as a result of adding the new variable to the model. The statistical significance of the obtained \(NRI\) value is checked with a dedicated Z test.

• Integrated Discrimination Improvement (\(IDI\)) – a coefficient showing the difference between the mean change in predicted probability (between the "old" and "new" model) in the group of objects experiencing the event and the group of objects that did not experience the event. As with \(NRI\), the statistical significance of \(IDI\) is checked with a dedicated Z test.

U-smile – the U-smile method described in the Logistic regression chapter can also be used to compare successive (neighbouring) models: the reference model is the earlier model on the list, and the new model is the next one. The report contains the \(rLR\), \(BA\) or \(RB\) coefficients (to choose from in the AUC-IDI-NRI window) for the four subclasses, the net coefficients for the classes, the overall coefficient, the shares \(I\) and the significance tests; a U-smile plot can be included for each pair. The \(p\) values of the tests assume nested models – for non-nested pairs (the all subsets method) they should be treated as indicative only.

In PQStat, model comparison can be carried out manually or automatically.

• Manual model comparison consists of building 2 models:

  • the full model – the model with more variables,
  • the reduced model – the model with fewer variables; such a model is obtained from the full model by removing the variables that, from the point of view of the phenomenon under study, are superfluous.

  The choice of independent variables in the compared models, and then the choice of the better model on the basis of the results of the comparison, is up to the researcher.

• Automatic model selection – the program builds successive models itself and indicates the best one according to the chosen criterion.

  Model terms. The unit that the automatic methods add to or remove from the model is a term: a single variable, a whole dummy variable (all its 0–1 columns at once) or an interaction. The hierarchy principle applies: an interaction can be in the model only when its main effects are in it, and only the interaction can be removed before them.

  The selection criterion is chosen in the analysis window:

  • AIC, AICc or BIC – information criteria; the model with the lower value of the criterion is better. A move (adding or removing a term) is accepted only when it lowers the criterion;
  • \(p\) of the nested test – the likelihood ratio test comparing the model before and after the change, with a given \(p\) threshold. A term is removed when \(p\) is greater than the threshold, and added when \(p\) is smaller than the threshold. The same \(p\) value is shown in the report as the comparison of neighbouring models;
  • validation error – the mean deviance determined on observations that did not take part in fitting the model (described below: Out-of-sample evaluation of models). A move is accepted only when it reduces this error. The criterion requires the Validation option to be switched on – choosing it while validation is off switches cross-validation on.

  Selection methods:

  • backward elimination – starts from the model with all the variables; in each step the term whose removal lowers the criterion the most is removed; it ends when no removal lowers the criterion any further;
  • forward selection – starts from the null model (intercept only); in each step the term that lowers the criterion the most is added; it ends when no addition lowers the criterion any further;
  • bidirectional stepwise – starts from the null model; in each step both additions and removals of terms are considered and the best move is chosen; it ends when no move improves the criterion;
  • all subsets – all models consistent with the hierarchy principle are fitted (a maximum of 15 terms), and the best model of each size is presented in the report; this method uses the information criteria or the validation error (the \(p\) criterion is not available in it).

  All the models are computed on the same set of observations – rows with missing values in any of the candidate variables are discarded once, before the analysis. Thanks to this, the information criteria, the likelihood ratio tests and the comparisons of ROC curves, \(NRI\) and \(IDI\) are comparable between models.

  The report contains: the selection method and criterion, the number of terms and of fitted models, the information criteria of the null model, the successive models of the selection path (each with the value of \(-2\ln L\), the AIC, AICc and BIC criteria, the \(R_{Pseudo}^2\), \(R^2_{Nagelkerke}\) and \(R^2_{Cox-Snell}\) coefficients, and the table of coefficients and odds ratios) and the comparison of each neighbouring pair of models with the likelihood ratio test (the \(\chi^2\) statistic, degrees of freedom, \(p\) value). If the evaluation of predictive value was selected in the analysis window, the comparison of ROC curves, \(NRI\) and \(IDI\) are also determined for each neighbouring pair of models – regardless of whether the models are nested. The best model according to the chosen criterion is marked in the report. If no term improves the null model, the report states that the null model is the best. With the all subsets method, neighbouring models may not be nested – in that case the likelihood ratio test is not performed.

  Note! The \(p\) values and confidence intervals in an automatically selected model are too optimistic, because the same data set was used both to select the variables and to estimate their effects. Automatic selection should be treated as an aid in searching for a model, and the final choice should be based on subject-matter knowledge. The selected model should be built again in the Logistic regression window and its quality assessed.

Out-of-sample evaluation of models

The fit measures, the information criteria and the tests, as well as the ROC curves, \(NRI\) and \(IDI\), are determined on the same observations on which the coefficients of the models were estimated. Such an assessment is too optimistic – a model usually predicts new cases worse than its fit would suggest, and the difference grows with the number of variables. For this reason, as in comparing multiple regression models, the analysis window makes it possible to evaluate the models on observations that did not take part in fitting them (the Validation option): \(k\)-fold cross-validation (each model is additionally fitted \(k\) times, each time without one of the \(k\) parts of the data, and evaluated on the part left out) or a random test set (the models are fitted on the training part and evaluated on the test part set aside). The split is random, determined by the Seed, and stratified by the dependent variable: the observations with and without the event are split separately, so that the percentage of events is similar in every part.

For each model the report then gives three additional measures:

• mean deviance (\(-2\ln L/n\)) – the value of \(-2\ln L\) determined from the probabilities predicted out of sample, divided by the number of evaluated observations. The lower it is, the better the model; the measure strongly „penalises” predictions that are confident but wrong. This is the validation error – the best model by validation is indicated according to it, and the validation error selection criterion is based on it;
• AUC – the area under the ROC curve built from the probabilities predicted out of sample; it describes the ability of the model to distinguish new objects with the event from objects without the event (0.5 – a useless model, 1 – perfect discrimination);
• Brier score – the mean squared difference between the predicted probability and the actual outcome (1 – event, 0 – no event)[29]; the lower it is, the better the predictions.

The point of reference is the null model, which assigns to every observation the frequency of the event from the training part – its mean deviance and Brier score are given in the model selection block of the report (the \(AUC\) of such a model is 0.5). The same block indicates the best model by validation – regardless of the selection criterion chosen. Out-of-sample evaluation also works in the manual comparison of models.

When validation is switched on, the comparison of the predictive value of the models (ROC curves, \(NRI\), \(IDI\), U-smile) is also determined from the probabilities predicted out of sample: in cross-validation for all the observations, and with a test set – for the observations of the test set. The report marks this with the row ROC, IDI, NRI, U-smile: validation predictions. These results then show whether the additional variables improve the prediction of new cases, and not only the fit to the data on which the model was built.

How to set up the analysis

Statistics \(\to\) Multivariate models \(\to\) Logistic regression – comparing models.

In the window we indicate the dependent variable \(Y\) with the value denoting the event, and the independent variables of the full and the reduced model (in automatic selection – the candidate variables). Model selection options:

• Type of comparison – manual or one of the automatic selection methods,
• Criterion – AIC, AICc, BIC, p (LR / F test) with the threshold given in the adjacent field, or validation error; in the manual mode the criterion is not used,
• Validation – none (evaluation on the training set only), k-fold CV, k = with the number of folds in the adjacent field (from 2 to 100, 10 by default) or test set, % = with the share of the test set as a percentage of the observations (from 5 to 90, 30 by default),
• Seed – the seed of the random split of the data (12345 by default); the same seed means the same split,
• AUC/U-smile – the choice of the comparisons of predictive value (ROC curves, \(NRI\), \(IDI\), U-smile) and of the related plots.

Performing and interpreting the analysis