Yes, we’ve seen this pattern where standard metrics look fine but trust in the model slowly erodes. In our case, the issue wasn’t a sudden performance drop but rather small inconsistencies in specific segments that kept creeping in. A few examples:

• Certain user groups or data patterns started showing slightly worse predictions over time

• Edge cases that used to be rare became more common, leading to more manual overrides

• The model’s outputs stopped aligning with expected business outcomes even though accuracy/AUC stayed steady

What helped us was adding deeper segment-level monitoring rather than relying on aggregate metrics alone. Once we looked at performance broken down by key cohorts or feature buckets, we could see that some segments were drifting earlier than the overall metric.

We also found that clear communication with stakeholders about what the metrics actually mean was important. Sometimes people lose trust not because the model is failing badly, but because they see inconsistencies that aren’t reflected in high-level numbers.