Function Repository Resource:

ConfusionMatrixToClassifierMeasurementsObject

Convert a confusion matrix into a ClassifierMeasurementsObject from which statistics can be derived

Contributed by: Seth J. Chandler

ResourceFunction["ConfusionMatrixToClassifierMeasurementsObject"][m,classes]

converts a numeric confusion matrix m into a ClassifierMeasurementsObject where classes are a list of the classes.

Details

If the confusion matrix contains more columns than rows, it will be assumed that the final column is for indeterminate cases; these indeterminate cases will not be considered when generating the ClassifierMeasurementsObject.

The input confusion matrix should follow the convention that the actual class is on the rows and the machine's classification is on the columns.

Examples

Basic Examples (2)

Create a ClassifierMeasurementsObject out of a confusion matrix where the classes are "healthy" and "sick":

In[1]:=

Out[1]=

Show the properties of the object generated by the confusion matrix:

In[2]:=

Out[2]=

Scope (3)

Show a confusion matrix in which the last column represents indeterminate cases and thus is dropped:

In[3]:=

Out[3]=

A confusion matrix with five classes:

In[4]:=

ResourceFunction[
"ConfusionMatrixToClassifierMeasurementsObject"][{{3, 4, 2, 1, 6}, {0, 8, 1, 2, 4}, {1, 2, 20, 3, 2}, {0, 0, 5, 11, 2}, {2, 5, 8, 0, 44}}, {"Abreu", "Altuve", "Alvarez", "Bregman", "Valdez"}]

Out[4]=

Numeric properties of a ClassifierMeasurementsObject are available:

In[5]:=

With[{numericProperties = {"Accuracy", "AccuracyBaseline", "AreaUnderROCCurve", "ClassRejectionRate", "CohenKappa", "ConfusionDistribution", "ConfusionFunction", "ConfusionMatrix", "DecisionUtilities", "Error", "F1Score", "FalseDiscoveryRate", "FalseNegativeNumber", "FalseNegativeRate", "FalsePositiveNumber", "FalsePositiveRate", "GeometricMeanProbability", "Likelihood", "MatthewsCorrelationCoefficient", "MeanDecisionUtility", "NegativePredictiveValue", "Perplexity", "Precision", "Probabilities", "Recall", "RejectionRate", "ScottPi", "Specificity", "TopConfusions", "TrueNegativeNumber", "TruePositiveNumber"}},
Dataset[
AssociationThread[numericProperties, ResourceFunction[
"ConfusionMatrixToClassifierMeasurementsObject"][{{13, 7}, {9, 71}}, {"healthy", "sick"}][numericProperties]
]]] // ResourceFunction["FormatDataset"][MaxItems -> 5, HeaderStyle -> 9, ItemStyle -> 9, HeaderSize -> 14]

Out[5]=

Properties that produce plots are also available:

In[6]:=

With[{plotProperties = {"AccuracyRejectionPlot", "ConfusionMatrixPlot"}},
Dataset[
AssociationThread[plotProperties, ResourceFunction[
"ConfusionMatrixToClassifierMeasurementsObject"][{{13, 7}, {9, 71}}, {"healthy", "sick"}][plotProperties]
]]]

Out[6]=

Applications (2)

Obtain a variety of statistics on a sample confusion matrix found in the Wikipedia entry for that topic:

In[7]:=

Out[7]=

In[8]:=

With[{properties = {"Accuracy", "CohenKappa", "PositivePredictiveValue" -> "cancer", "Sensitivity" -> "cancer",
"Specificity" -> "cancer"}}, AssociationThread[properties, wikiCM[properties]]] // Dataset

Out[8]=

Perform a Bayesian analysis with the data by computing the probability that a person who tests negative for cancer really does not have cancer:

In[9]:=

Out[9]=

Examine the accuracy and "ROCCurve" of a classifier in which one uses sex as a basis for determining if someone would survive the sinking of the Titanic:

In[10]:=

Normal[Query[Values, Values][
Query[GroupBy[If[#survived, "survived", "died"] &]/*KeySort, GroupBy[If[#sex == "female", "survived", "died"] &]/*KeySort, Length][ExampleData[{"Dataset", "Titanic"}]]]]

Out[10]=

In[11]:=

Out[11]=

Possible Issues (5)

Some properties of a ClassifierMeasurementsObject will be Missing:

In[12]:=

Out[12]=

In[13]:=

Out[13]=

The "ClassMeanCrossEntropy" property will take on infinite values:

In[14]:=

Out[14]=

Properties that yield show examples will not be well defined:

In[15]:=

Out[15]=

If a row of the confusion matrix sums to zero, i. e. there are no examples of a certain class, an error message will be generated:

In[16]:=

Out[16]=

The function will work, however, if it has positive row sums but a column sum of the matrix is zero:

In[17]:=

Out[17]=

Neat Examples (4)

Create a composite confusion matrix from data that has been analyzed using cross-validation and determine the performance of the "NaiveBayes" classifier:

In[18]:=