In production-grade *machine learning workflows*, feature engineering is *not a one-time task* — it’s part of a *continuous lifecycle*, especially when facing *data drift*. Here’s how it’s typically approached:

—

*How Often Feature Engineering is Revisited*

– *Continuously monitored*, but *actively revisited*:
– *On schedule*: Every 1–3 months in stable systems.
– *On trigger*: Immediately when key indicators show drift or performance drop.
– *After new data sources or business logic changes.*

—

*Key Indicators That Signal Feature Re-Evaluation*

1. *Model Performance Degradation*
– Drop in metrics like accuracy, precision, recall, F1, AUC, etc.
– Increasing prediction error (MAE, RMSE, log loss).

2. *Data Drift / Concept Drift*
– *Statistical drift* in feature distributions (e.g., using KS test, Jensen–Shannon divergence).
– Real-world meaning of features changes (concept drift).

3. *Target Drift*
– Changes in target label distribution over time.

4. *Feature Importance Shifts*
– Features that were once predictive lose value.
– New features become more relevant.

5. *Pipeline Failure or Latency*
– Real-time features break due to upstream schema/API changes.
– Feature generation becomes too slow or costly.

*Monitoring Strategies*

– *Feature Store Tracking* (e.g., Feast, Tecton):
– Track feature statistics over time.
– Compare training vs. live data distributions.

– *Drift Detection Tools*:
– Tools like EvidentlyAI, WhyLabs, or custom dashboards with alerts.
– Use metrics like Population Stability Index (PSI), Data Stability Index (DSI), KL divergence.

– *Shadow Deployment*:
– Deploy new feature pipelines/models in parallel for testing against live data before switching.

– *Canary Models / Champion-Challenger*:
– Compare old vs. new models using same inputs to detect divergence.

—

*Bottom Line*

Revisit feature engineering:
– *Proactively*: every 1–3 months or with new data sources.
– *Reactively*: when monitoring flags drift, performance drops, or feature behavior changes.