How to prevent overfitting in machine learning models

Short answer

Overfitting happens when a model memorizes training data instead of learning generalizable patterns. Prevent it by simplifying the model, adding more data, using regularization, and validating properly.

Steps

1. Split data into training, validation, and test sets using stratified sampling.
2. Apply L1 or L2 regularization to penalize large coefficients and reduce model complexity.
3. Use dropout layers in neural networks to randomly deactivate neurons during training.
4. Implement early stopping based on validation loss to halt training before memorization begins.
5. Augment training data with transformations that preserve labels to artificially expand the dataset.

Tips

• Monitor training and validation curves; a growing gap indicates overfitting.
• Start with simpler models before moving to complex architectures.
• Use ensemble methods like bagging and boosting to improve generalization.
• Cross-validate results across multiple folds to ensure stability.

Common issues

• Using too many parameters relative to the number of training examples.
• Training for too many epochs without validation checks.
• Leaking test data into training or validation pipelines.
• Ignoring data quality issues that force the model to fit noise.

Example

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(784, 256),
    nn.ReLU(),
    nn.Dropout(0.2),
    nn.Linear(256, 10)
)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

This snippet defines a simple neural network with dropout for regularization, a cross-entropy loss, and the Adam optimizer in PyTorch.