How to evaluate a classification model

Short answer

Classification models are evaluated using a combination of threshold-dependent and threshold-independent metrics that capture different aspects of prediction quality.

Steps

1. Build a confusion matrix to count true positives, true negatives, false positives, and false negatives.
2. Calculate precision to measure positive prediction reliability and recall to measure coverage of actual positives.
3. Compute the F1 score as the harmonic mean of precision and recall for imbalanced datasets.
4. Plot the ROC curve and calculate AUC to assess performance across all thresholds.
5. Review the precision-recall curve when dealing with highly imbalanced classes.

Tips

• Do not rely solely on accuracy when classes are imbalanced.
• Use Cohen's kappa or Matthews correlation coefficient for balanced insight.
• Compare metrics on both training and validation sets to detect overfitting.
• Report confidence intervals when presenting metrics to stakeholders.

Common issues

• Optimizing for accuracy alone on skewed datasets produces misleading results.
• Choosing the wrong threshold for the business cost of false positives versus false negatives.
• Evaluating on the same data used for training or hyperparameter tuning.
• Ignoring calibration when probability outputs are needed for decision-making.

Example

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(784, 256),
    nn.ReLU(),
    nn.Dropout(0.2),
    nn.Linear(256, 10)
)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

This snippet defines a simple neural network with dropout for regularization, a cross-entropy loss, and the Adam optimizer in PyTorch.