How to tune hyperparameters effectively

Short answer

Hyperparameter tuning finds the configuration that maximizes model generalization by systematically exploring the search space of possible settings.

Steps

1. Define the hyperparameter search space based on domain knowledge and literature.
2. Start with a coarse random search to identify promising regions quickly.
3. Use Bayesian optimization to model the objective and guide efficient exploration.
4. Apply nested cross-validation to obtain unbiased estimates of tuned model performance.
5. Validate the final chosen hyperparameters on the held-out test set.

Tips

• Use log-scale sampling for learning rates and regularization strengths.
• Limit the search space to avoid overfitting the validation data.
• Parallelize evaluations when compute resources are available.
• Track all experiments with tools like MLflow or Weights and Biases.

Common issues

• Grid search becomes computationally infeasible in high-dimensional spaces.
• Running too many trials leads to implicit overfitting on the validation set.
• Ignoring interaction effects between hyperparameters.
• Failing to fix the random seed makes results non-reproducible.

Example

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(784, 256),
    nn.ReLU(),
    nn.Dropout(0.2),
    nn.Linear(256, 10)
)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

This snippet defines a simple neural network with dropout for regularization, a cross-entropy loss, and the Adam optimizer in PyTorch.