HomeTechniques and TipsNeuralToolsInterpreting Incorrect% in Testing Report

15.21. Interpreting Incorrect% in Testing Report

My question is about the Detailed Report that NeuralTools produces when testing category predictions in a PNN. With a Good result, the Incorrect% is 100 minus the Prediction%, which makes sense to me. But with a Bad result, there doesn't seem to be a relationship between the Incorrect% and the Prediction%. What does the Incorrect% mean?

The Incorrect% is the sum of the probabilities for all the incorrect categories.

This is easier to understand if you "change Detailed Report settings to show the probabilities assigned by a Probabilistic Neural Net to every possible category for the dependent variable." Click Utilities » Application Settings. In the Reports section, click the drop-down menu at the end of the Columns in Detailed Reports row. A dialog box titled NeuralTools — Columns to Display in Detailed Reports will open. "In that dialog, select Probabilities of All Categories (for PNN) for Testing. Then train a PN Net on a data set with at least 3 categories in the dependent variable." (Source: NeuralTools 6 user manual, page 48.)

Attached is part of the results from our standard example "Auto Loans 1a" (Help » Example Spreadsheets) with that application setting in effect. (Click the image to open it full size, in a new tab.)

Detailed Testing Report — click to enlarge.

Row 5 is probably the clearest: The correct answer was "late payments", but NeuralTools had computed there was only a 43.12% chance it was correct. NeuralTools picked "timely payments" because it thought there was a 54.69% chance that was correct. It computed a 2.19% probability for the third category, "default". The Incorrect% is 100% minus the probability of the correct answer, 100–43.12 = 56.88%. That same 56.88% is the sum of the probabilities of the incorrect answers, 54.69+2.19%. By contrast, for a Good result like row 6, Incorrect% is always 100% minus Prediction%, since the predicted answer was correct and all the incorrect answers were also not predicted.

For Good and Bad answers alike, Incorrect% is the total probability of all incorrect answers, which equals 100 minus the correct%. However, NeuralTools displays the Prediction%, not the correct%. If the answer was Good, the predicted answer was the correct answer, so the probability shown in Prediction% is the correct%. But for Bad answers, the prediction was not the correct answer; therefore, the correct% is not shown, the Prediction% is the probability of the one incorrect answer that had highest probability, and the Incorrect% is the total probability of all the incorrect answers.

Last edited: 2015-05-08

This page was: Helpful | Not Helpful