We’ve tried collaborative filtering, not working. Anyone solved this at scale?
We’ve tried collaborative filtering, not working. Anyone solved this at scale?
Apply hard business rules and an exploration layer. Don’t let raw math ruin the UX. Implement immediate post-processing filters to block out-of-stock, irrelevant, or repetitive categories. Additionally, inject a 10% mix of globally trending or popular items to break bad feedback loops and help the model safely discover new user interests.
Upgrade to a Graph Neural Network (GNN) and tune your weights. Standard matrix factorization misses complex, multi-hop user-item relationships. Transitioning to a GNN (like LightGCN) will better capture deep structural similarities. Also, ensure your loss function downweights superficial clicks and heavily rewards high-value interactions like purchases or long dwell times.
Switch to a Hybrid Retrieval/Ranking system. Pure collaborative filtering fails at scale due to data sparsity. Use it only for the initial retrieval step to get a broad pool of items. Then, pass those items through a second ranking layer (like XGBoost or a Deep & Cross Network) that filters and scores them based on real-time metadata, user context, and categories.