AI & ML
CascadeRec: Deployed Adaptive LLM Compute Allocation for Multi-Tenant Content Recommendation
arXiv cs.IR0 views
arXiv:2611.08566v1 Announce Type: new Abstract: Large language models (LLMs) are attractive rerankers for recommendation. Running an expensive model on every request is costly, and for users with sparse interaction histories it often lowers quality. We present CascadeRec, a two-tier LLM reranking cascade deployed in production at a large financial planning and analysis (FP&A) SaaS provider. It serves content recommendations to enterprise tenants on AWS EKS and GKE. A lightweight Tier-1 model reranks every request. A router escalates only the requests where extra inference-time compute is likely to help, and sends them to a more expensive Tier-2 procedure (self-consistency sampling, multi-path reasoning, or a larger model). The system has run in a production A/B experiment for over a month. Compared with uniform use of the expensive tier, CascadeRec has reduced inference cost by about 10%, with minimal latency impact and no quality regression observed to date. Tenant data cannot be released, so we also evaluate the pipeline on public benchmarks (MovieLens-1M and Amazon-Electronics). Segment-level analysis with paired-bootstrap confidence intervals shows that uniform LLM reranking lowers NDCG@10 for sparse-history users by up to 30%, and its only statistically significant gain appears on the segment with the longest histories. Escalating cold users can make the loss larger instead of repairing it. Which segments benefit from escalation depends on the domain, and this motivates the segment-gated rollout we use in production. We describe the deployment architecture, the measured benefits, and lessons learned, including inter-tenant load imbalance as a driver for cascading and the handling of LLM output-parsing failures at the serving layer.
Source:https://arxiv.org/abs/2611.08566PreviousRedMall: Generative Recommendation for Large-Scale E-commerceNextSteerGR: Steering Generative Recommendation via Sparse Autoencoders for Controllable Game Discovery
Related posts
The Midterms Were a Pretty Good Sign for the Right-to-Repair Movement
0Wired
New Hulu Doc ‘Get Huge’ Shows the Deadly Toll of Looksmaxxing
0Wired

Instagram’s AI age detection is expanding globally
0The Verge
Expanding Teen Account Protections to Teens Around the World
0Meta Newsroom
Tối ưu hóa hiệu suất đầu ra: Tiền vệ Njie vực dậy hàng công Fiorentina với 5 lần in dấu giày
0dev.to Data
Hiệu suất săn bàn ấn tượng của Gouiri thúc đẩy chuỗi toàn thắng của Marseille
0dev.to Data