How to apply transfer learning in deep learning

Short answer

Transfer learning accelerates model development by reusing representations learned from large datasets and adapting them to a target task with limited data.

Steps

1. Select a pretrained model trained on a source dataset related to your domain.
2. Replace the final classification layer to match the number of classes in your target task.
3. Freeze all pretrained layers and train only the new head for a few epochs.
4. Gradually unfreeze deeper layers and fine-tune with a low learning rate.
5. Evaluate on a validation set and stop when performance plateaus.

Tips

• Use smaller learning rates for pretrained layers to avoid catastrophic forgetting.
• Keep batch normalization layers in evaluation mode during initial training if data is scarce.
• Choose deeper models only when the target dataset is large enough to benefit.
• Compare feature extraction versus fine-tuning to determine the best strategy.

Common issues

• Domain mismatch between source and target data reduces transfer effectiveness.
• Fine-tuning too aggressively destroys useful pretrained features.
• Overfitting when the target dataset is much smaller than the source dataset.
• Class imbalance in the target dataset biasing the new classification head.

Example

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(784, 256),
    nn.ReLU(),
    nn.Dropout(0.2),
    nn.Linear(256, 10)
)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

This snippet defines a simple neural network with dropout for regularization, a cross-entropy loss, and the Adam optimizer in PyTorch.