I’m working on a system where numerical values inside content are updated dynamically. The issue is that the title often contains the same number, and when the value changes in the content, the title becomes inaccurate.

I’m looking for a reliable way to automatically detect these changes and update the title accordingly without manually editing each post. Has anyone solved a similar problem or found a scalable approach for this?

I came across this pipeline setup where feature engineering is being added before a ColumnTransformer, but the new features don’t seem to flow correctly through the pipeline:

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.base import BaseEstimator, TransformerMixin

class FeatureAdder(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        return self
    
    def transform(self, X):
        X['new_feature'] = X['col1'] * X['col2']
        return X

pipeline = Pipeline([
    ('feature_add', FeatureAdder()),
    ('preprocess', ColumnTransformer([
        ('num', StandardScaler(), ['col1', 'col2']),
        ('cat', OneHotEncoder(), ['col3'])
    ]))
])

The issue is:

• The newly created new_feature is not included in the ColumnTransformer

• This leads to it being dropped during transformation

In a setup like this:

• Should the ColumnTransformer be dynamically updated to include new features?

• Or is it better to handle feature engineering outside the pipeline altogether?

• How do you ensure feature consistency without breaking pipeline modularity?

Many models perform well during training and validation but start degrading in production due to data drift, concept drift, or changing user behavior. What monitoring strategies, retraining pipelines, or evaluation practices do you use to maintain model performance in production environments?

I’m working on a data science project with time-ordered data, and I’m seeing a significant drop in model performance once I move from training to validation. I’m sharing a simplified version of the problem and code below.

The dataset represents events over time, and the target is binary. I initially used a random train-test split, but later switched to a time-based split to better reflect real-world usage. After this change, performance dropped sharply, and I’m trying to understand whether this is expected or if I’m doing something wrong.

Here’s a simplified version of the code:

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score

# sample data
df = pd.read_csv("data.csv")
df = df.sort_values("event_time")

X = df.drop(columns=["target"])
y = df["target"]

# time-based split
split_index = int(len(df) * 0.8)
X_train, X_test = X.iloc[:split_index], X.iloc[split_index:]
y_train, y_test = y.iloc[:split_index], y.iloc[split_index:]

model = RandomForestClassifier(random_state=42)
model.fit(X_train, y_train)

preds = model.predict_proba(X_test)[:, 1]
print("AUC:", roc_auc_score(y_test, preds))

With a random split, the AUC was around 0.82.
With the time-based split, it drops to around 0.61.

I’m trying to understand:

• Is this performance gap a common sign of data leakage in the original setup?

• Are tree-based models like Random Forests particularly sensitive to temporal shifts?

• What are good practices to diagnose whether this is concept drift, feature leakage, or simply a harder prediction problem?

• Would you approach validation differently for time-dependent data like this?

Looking for general guidance, validation strategies, or patterns others have seen in similar scenarios.