PQStat Validation and prediction for multiple regression
When to use:
Validation of a model is a check of its quality. It is first performed on the data on which the model was built (the so-called training set), that is, it is returned in the report describing the resulting model. In order to be able to judge with greater confidence how suitable the model is for forecasting new data, an important part of validation is applying the model to data that were not used in estimating the model. If the summary based on the training data is satisfactory, i.e. the calculated errors, \(R^2\) coefficients, and information criteria are at a satisfactory level, and the summary based on the new data (the so-called test set) is equally favourable, then it can be concluded with high probability that such a model is suitable for prediction. The test data should come from the same population from which the training data were selected. It is often the case that before building a model we collect data, and then randomly divide it into a training set, i.e. the data that will be used to build the model, and a test set, i.e. the data that will be used for additional validation of the model.
Prediction – most often, the final step in regression analysis is to use the constructed and previously validated model for prediction.
Definitions of measures
The estimated value is calculated with a certain error. Therefore, in addition, limits resulting from the error are determined for the value predicted by the model:
- for the expected value, confidence limits are determined,
- for a single point, prediction limits are determined.
How to set up the analysis
Advanced statistics \(\to\) Multivariate models \(\to\) Multiple regression – prediction/validation.
To perform validation or prediction, it is necessary to indicate the model on the basis of which we want to perform it. This can be done on the basis of:
- a multiple regression model built in PQStat – simply select a model from those assigned to the given sheet, and the number of variables and model coefficients will be set automatically; the test set should be in the same sheet as the training set;
- a model not built in PQStat but obtained from another source (e.g. described in a scientific paper we have read) – in the analysis window, enter the number of variables and the coefficients for each of them.
In the analysis window, indicate the new variables that should be used for validation or prediction. Prediction for a single object can be performed together with building the model, that is, in the analysis window Advanced statistics\(\to\)Multivariate models\(\to\)Multiple regression, while prediction for a larger group of new data is performed via the menu Advanced statistics\(\to\)Multivariate models\(\to\)Multiple regression – prediction/validation.
Performing and interpreting the analysis
Continuation of Example 1 (publisher.pqs file): to predict gross profit from book sales, the publisher built a regression model based on a training set with item 16 removed (i.e. 39 books). The model included: production costs, advertising costs, and author's popularity (1=popular author, 0=not). We will build this model once again based on the training set, and then, to make sure the model will work correctly, we will validate it on a test data set. If the model passes this test, we will use it for prediction for new book titles. To use the appropriate sets, we set a data filter each time.
For this purpose, we will use the data for three new book titles added at the end of the set. We select the Prediction option, set the filter to the new data set, and use our model to predict the gross profit for these books.
It turns out that the highest gross profit (between 64 and 85 thousand dollars) is predicted for the first book – the most heavily advertised and most expensively produced book by a popular author.