AI & ML
Set-Valued Routing for Language-Model Agent Systems: A WildChat Benchmark and Cost-Aware Evaluation
arXiv cs.IR0 views
arXiv:2606.28925v3 Announce Type: replace-cross Abstract: Language-model agent systems must map natural-language requests to one or more specialized agents before planning or tool execution. This decision is often treated as top-1 selection, although compositional requests can require several capabilities and unnecessary dispatch raises cost. We formulate fixed-catalog routing as set-valued language understanding and construct a benchmark of 3,000 WildChat prompts over 12 capability-defined agents. Reproducible schema-guided reference sets are balanced by agent frequency and route cardinality. We compare profile retrieval, multilabel classifiers, a fine-tuned text encoder, and catalog-conditioned zero-shot GPT-4o. Evaluation combines set quality, latency, capability coverage, and ordinal-cost utility. The encoder attains 89.59 F1, compared with 71.79 for linear SVM and 41.51 for GPT-4o, and retains 88.02 F1 under a harder semantic-group split. We further introduce Weighted Agent Routing (WAR), a deterministic post-scoring rule that changes the quality-cost operating point without retraining; Encoder+WAR obtains the best constrained utility. The results show that catalog-specific supervision and explicit set construction are important for multi-agent dispatch, while cost-aware decision layers can complement strong language encoders.
Source:https://arxiv.org/abs/2606.28925PreviousAgentSelect: Benchmark for Narrative Query-to-Agent RecommendationNextWhere Does Retrieval-Based Open-Ended Evaluation Fail? Automatic Taxonomy Induction from Long-Form Medical Answer Factuality Verification
Related posts
Inter-AI Morality: How AI Systems Treat One Another [pdf]
0Hacker News New
Solar plane soars to 47,600 ft, on target to fly 100 hours
0Hacker News New
Why you shouldn't pour coffee down the drain
0Hacker News New
An internet radio station run by AI
0Hacker News New
APM – A dependency manager for AI agents
0Hacker News New
AI Source Code Analysis Detection Evasion Techniques
0Hacker News New