AI & ML
CascadeRec: Deployed Adaptive LLM Compute Allocation for Multi-Tenant Content Recommendation
arXiv cs.IR0 views
arXiv:2611.08566v1 Announce Type: new Abstract: Large language models (LLMs) are attractive rerankers for recommendation. Running an expensive model on every request is costly, and for users with sparse interaction histories it often lowers quality. We present CascadeRec, a two-tier LLM reranking cascade deployed in production at a large financial planning and analysis (FP&A) SaaS provider. It serves content recommendations to enterprise tenants on AWS EKS and GKE. A lightweight Tier-1 model reranks every request. A router escalates only the requests where extra inference-time compute is likely to help, and sends them to a more expensive Tier-2 procedure (self-consistency sampling, multi-path reasoning, or a larger model). The system has run in a production A/B experiment for over a month. Compared with uniform use of the expensive tier, CascadeRec has reduced inference cost by about 10%, with minimal latency impact and no quality regression observed to date. Tenant data cannot be released, so we also evaluate the pipeline on public benchmarks (MovieLens-1M and Amazon-Electronics). Segment-level analysis with paired-bootstrap confidence intervals shows that uniform LLM reranking lowers NDCG@10 for sparse-history users by up to 30%, and its only statistically significant gain appears on the segment with the longest histories. Escalating cold users can make the loss larger instead of repairing it. Which segments benefit from escalation depends on the domain, and this motivates the segment-gated rollout we use in production. We describe the deployment architecture, the measured benefits, and lessons learned, including inter-tenant load imbalance as a driver for cascading and the handling of LLM output-parsing failures at the serving layer.
Source:https://arxiv.org/abs/2611.08566PreviousRedMall: Generative Recommendation for Large-Scale E-commerceNextSteerGR: Steering Generative Recommendation via Sparse Autoencoders for Controllable Game Discovery
Related posts
W-2 vs 1099 Calculator: A Worked Offer Comparison for 2026
0dev.to

I tested ChatGPT's PDF Content Credentials: what survives a re-save, a print and an edit
0dev.to
I Built ResearchLens AI to Make Data Exploration and ML Benchmarking Easier
0dev.to
Fold Epic Paper Airplanes
0Hacker News
The Midterms Were a Pretty Good Sign for the Right-to-Repair Movement
0Wired
New Hulu Doc ‘Get Huge’ Shows the Deadly Toll of Looksmaxxing
0Wired