This happens because token-level loss does not capture sentence-level quality. Transformers are trained to predict the next token, not to produce coherent or accurate full sequences. A model can become very good at predicting individual words while still producing poor translations.

Loss measures how well each token matches the reference, but translation quality depends on word order, fluency, and semantic correctness across the entire sequence. These properties are not directly optimized by standard cross-entropy loss.

Using better decoding strategies such as beam search, label smoothing, and sequence-level evaluation helps align training with actual quality. In some setups, reinforcement learning or minimum-risk training is used to optimize sequence metrics directly.