This usually happens when feedback loops in production reinforce certain predictions more than others.

In many real systems, model outputs influence the data collected next. If one class is shown or acted upon more often, future training data becomes skewed toward that class. Over time, the model appears to “prefer” it, even if the original distribution was balanced.

To fix this, monitor class distributions in both predictions and incoming labels. Introduce sampling or reweighting during retraining so minority classes remain represented. In some systems, delaying or decoupling feedback from training helps break the loop.

Common mistakes:

Assuming bias only comes from training data. Retraining on production data without auditing it or monitoring accuracy but not class balance

Models don’t just learn from data — they learn from the systems around them.