When a transformer collapses to a single prediction, it’s almost always due to a training signal problem rather than model architecture.

This happens if gradients are vanishing, labels are incorrectly encoded, or the loss function doesn’t match the task. For example, using CrossEntropyLoss with already-softmaxed outputs will silently break learning.

Start by checking that your labels vary and are correctly mapped. Then confirm that your final layer outputs raw logits and not probabilities. Run a single batch through the model and inspect gradient norms—if they’re near zero, learning isn’t happening.

Common mistakes:

• Using the wrong loss for multi-class vs multi-label tasks

• Forgetting to unfreeze pretrained layers

• Training with a learning rate that’s too low to escape initialization bias

If predictions are identical after thousands of steps, stop training and validate your data pipeline before changing the model.

In fine-tuning scenarios, also confirm that layers aren’t frozen unintentionally. Many pretrained checkpoints load with frozen encoders by default.