This happens because we train models in a perfect bubble and deploy them into a messy world. During training, the data is clean, rules are fixed, and nothing unexpected shows up. But once the model goes live, it has to rely on real production systems APIs, logs, user events that were built for operations, not for ML stability.

In production, data breaks quietly. Fields go missing, formats change without warning, timestamps lag, and edge cases start appearing. On top of that, real users don’t behave like historical users. Their preferences shift, traffic patterns change, and new behaviors emerge. None of this crashes the system immediately, so it’s easy to miss but the model slowly drifts away from the environment it was trained in.