I fine-tuned a pretrained Transformer on a small custom dataset.
Training finishes without errors.
But the generated outputs look random and off-topic.
It feels like the model forgot everything.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
I fine-tuned a pretrained Transformer on a small custom dataset.
Training finishes without errors.
But the generated outputs look random and off-topic.
It feels like the model forgot everything.
This happens because the model is overfitting and catastrophically forgetting pretrained knowledge.
When fine-tuning on small datasets, the Transformer’s weights drift away from what they originally learned. Use a lower learning rate and freeze early layers:
for param in model.base_model.parameters():
param.requires_grad = False
Also use weight decay and early stopping.
Common mistakes:
Learning rate too high
Training all layers on tiny datasets
No regularization
The practical takeaway is that pretrained models need gentle fine-tuning, not aggressive retraining.