This kind of drop is common when moving from random splits to time-based splits and often indicates that the original setup benefited from leakage or unrealistically easy correlations. Random splits allow the model to see future patterns indirectly, which inflates performance.

Tree-based models can struggle when feature distributions shift over time, so it’s worth checking feature drift and target stability. Monitoring feature importance changes and score distributions can help confirm this.

In most cases, the time-based result is the more honest signal. From there, techniques like rolling validation, feature decay, or retraining schedules usually matter more than model choice.