Offline metrics often fail to capture real user behavior.

In production, user interactions introduce feedback loops, latency constraints, and distribution shifts that static datasets don’t reflect. A model may optimize for offline accuracy but degrade user experience.

Instrument live metrics and analyze segment-level performance. Often the failure is localized to specific cohorts or edge cases.

Common mistakes:

1. Relying on a single offline metric

2. Ignoring latency and timeouts

3. Deploying without gradual rollout

Offline success is necessary but never sufficient.