Basic Examples (2)
Get some simple training and test data:
Run a classifier and create a ClassifierMeasurementsObject:
Create the ConfusionMatrixTrajectoryFunction:
Get the training and test sets of the Fisher iris data:
Create a classifier on the training set:
Construct a ClassifierMeasurementsObject from the classifier and the test set:
Construct a function that will, for any given probability threshold, construct the confusion matrix when "setosa" is designated as the positive class:
Apply that function to a threshold of 0.62 to get a confusion matrix:
Create a plot showing a trajectory of the false positive rate and true positive rate of the classifier for different thresholds:
Create a confusion matrix trajectory function as before, but let "virginica" be designated as the positive class:
Get the confusion matrix if the threshold is 0.62:
Scope (2)
One does not need a ClassifierMeasurementsObject to create the requisite function; the function will work with an Association that has the requisite key-value pairs as shown in the following example:
One can use the function on a neural network that has been converted to a classifier. Create a trained network following the method set forth here:
Create a ClassifierMeasurementsObject using testData:
Create the confusion matrix trajectory function with "versicolor" as the positive class:
Show the three-dimensional ROC (receiver operating characteristic) curve for the network:
Applications (2)
Create functions that compute the false positive rate and true positive rate from a confusion matrix:
Construct a three-dimensional receiver operating curve that shows the false positive rate and true positive rate for "good" wine as a function of the threshold probability required before the classifier calls a wine "good":
Construct a ClassifierMeasurementsObject from the classifier and the test set:
Create the confusion matrix function:
Create a three-dimensional plot:
View the graphic from the top as a traditional receiver operating curve:
The function works well in conjunction with the CrossValidateModel resource function and permits one to smooth the confusion matrix trajectory. Start this process by creating a query that will convert continuous wine quality values into discrete bins of Good, Bad and Mediocre:
Create wine data suitable for Classify:
Run the CrossValidateModel resource function to create five pairs of ClassifierFunction and ClassifierMeasurementsObject for each of the five folds:
Create confusion matrix trajectory functions for each of the five folds:
Plot the accuracy of the classifier over the threshold levels for each of the folds:
Create a composite view of the five accuracy plots:
Now create a function that computes confusion matrices for all five folds:
Plot the blended accuracy curve:
Show the blended accuracy plot and the components creating it:
Neat Examples (4)
Compare how a classifier performs on wines with low pH versus wines with high pH:
Show the three-dimensional ROC curves and draw points where the threshold is 0.5:
Show the graphic from the top and notice that even though the ROC curves look roughly the same in two dimensions, the false positive and true positive rates for the two different pH groupings of wine differ significantly at a threshold of 0.5:
Create the same plot but make the hue of each plot line depend on the accuracy of the classifier at that threshold: