I’m currently working on a production ML project, so I can’t share specific details about the domain or data.
We have a deployed model where performance looks stable in offline evaluation, but in real usage we suspect gradual drift. The challenge is that reliable ground truth only becomes available weeks or months later, which makes continuous validation difficult.
I’m trying to understand practical approaches teams use in this situation:
Looking for general patterns, tooling approaches, or lessons learned rather than domain-specific solutions.
Avatar Here’s a slightly casual, clear answer you can use:
Most teams don’t wait for ground truth to arrive. They monitor input and data drift, proxy metrics, and business signals to spot issues early. When labels finally come in, they use them for back-testing, retraining, and recalibration rather than immediate fixes.
The idea is to manage risk while learning, not to pause decisions until the data is perfect.
Most teams don’t wait for perfect ground truth. They rely on early signals like data drift, input distribution changes, and proxy business metrics to flag risk. When ground truth arrives late, it’s used for periodic back-testing, recalibration, and retraining rather than real-time correction.
In practice, teams combine delayed labels with monitoring, human review for edge cases, and clear retraining triggers. The goal is not to eliminate drift entirely, but to detect it early and control its impact until reliable feedback becomes available.
This kind of drop is common when moving from random splits to time-based splits and often indicates that the original setup benefited from leakage or unrealistically easy correlations. Random splits allow the model to see future patterns indirectly, which inflates performance.
Tree-based models can struggle when feature distributions shift over time, so it’s worth checking feature drift and target stability. Monitoring feature importance changes and score distributions can help confirm this.
In most cases, the time-based result is the more honest signal. From there, techniques like rolling validation, feature decay, or retraining schedules usually matter more than model choice.
