Neural networks often have narrow stability windows for learning rates.

A small increase can push updates beyond the region where gradients are meaningful, especially in deep or transformer-based models. This causes loss to explode or become NaN within a few steps.

Rollback to the last stable rate and introduce a scheduler instead of manual tuning. Warm-up schedules are especially important for large models.

Also verify that mixed-precision training isn’t amplifying numerical errors.

Common mistakes:

• Using the same learning rate across architectures

• Disabling gradient clipping

• Increasing rate without adjusting batch size

When in doubt, stability beats speed.