15.36. Calculation and Use of Variable Impacts

The purpose of Variable Impact Analysis is to measure the sensitivity of net predictions to changes in independent variables. This analysis is only done on training data. As a result of the analysis, every independent variable is assigned a "Relative Variable Impact" value; these are percent values and add to 100%. The lower the percent value for a given variable, the less that variable affects the predictions. The results of the analysis can help in the selection of a new set of independent variables, one that will allow more accurate predictions. For example, a variable with a low impact value can be eliminated in favor of some new variable.

However, one needs to keep in mind that the results of the Impact Analysis are relative to a given net. The fact that one net "learned" to disregard a given variable makes it likely that another net will also "learn" to disregard it; but then again, another training session with a different type of net might "discover" how to the variable can make a significant contribution to accurate predictions. In data sets with smaller numbers of cases and/or larger numbers of variables, the differences in the relative impact of the variables between trained nets may be more pronounced. Also, it is important to remember that these values are "relative". Suppose that with two independent variables one is assigned 99%, and the other 1%. This means that the latter is much less important than the former, but does not mean that it is unimportant, particularly if high accuracy of predictions is desired.

Only the training data set is included in the analysis. (If Auto-Testing or Auto-Prediction are used, those cases are not included. The reason is that they might have numeric values outside the training range, which could make analysis results more unpredictable.)

For a given category independent variable, for every case the analysis steps through all the valid categories for that variable, and measures the change to the predicted value. (With category prediction there is no numeric predicted value, but there are raw numeric net outputs on which the category prediction is based; those numeric outputs are used by the analysis.)

For a given numeric independent variable, for every case the analysis steps through the range from the minimum to the maximum training value for that variable, measuring the change to the predicted value (or, in the case of category prediction, change to the raw numeric outputs).

The internal details of the method are not crucial, because the purpose of that analysis is limited. It's not meant to support firm conclusions, like stating with high confidence that a given variable is irrelevant. Instead, it's meant to help in a search for the best set of independent variables: the results of the analysis may be telling us that a given variable looks irrelevant, sufficiently so that it's worth trying to train a net without this variable.