Token misalignment usually comes from mismatched tokenizers or improper handling of special tokens.

This happens when training and inference use different tokenizer versions or settings. Even a changed vocabulary order can shift outputs.

Always load the tokenizer from the same checkpoint as the model. When post-processing outputs, account for padding, start, and end tokens explicitly.

Common mistakes:

1. Rebuilding tokenizers manually

2. Ignoring attention masks

3. Mixing fast and slow tokenizer variants

Tokenizer consistency is non-negotiable in transformer pipelines.