AI & ML
MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents
arXiv cs.MA0 views
arXiv:2503.01935v2 Announce Type: replace Abstract: Large Language Models (LLMs) have shown remarkable capabilities as autonomous agents, yet existing benchmarks either focus on single-agent tasks or are confined to narrow domains, failing to capture the dynamics of multi-agent coordination and competition. In this paper, we introduce MultiAgentBench, a comprehensive benchmark designed to evaluate LLM-based multi-agent systems across diverse, interactive scenarios. Our framework measures not only task completion but also the quality of collaboration and competition using novel, milestone-based key performance indicators. Moreover, we evaluate various coordination protocols (including star, chain, tree, and graph topologies) and innovative strategies such as group discussion and cognitive planning. Notably, gpt-4o-mini reaches the average highest task score, graph structure performs the best among coordination protocols in the research scenario, and cognitive planning improves milestone achievement rates by 3%. Code and datasets are public available at https://github.com/MultiagentBench/MARBLE.
Source:https://arxiv.org/abs/2503.01935PreviousCBS with Continuous-Time RevisitNextUncertainty, bias and the institution bootstrapping problem
Related posts
A Matching Lower Bound for Online Smooth Calibration
0arXiv stat.ML
How Path Integral Diffusion Decides a Finite Spin Glass
0arXiv stat.ML
Prescriptive Learning of Linear Policies with Feasibility Guarantees for Binary Decision-Making
0arXiv stat.ML
IQS-BO: In-Context Query Selection for Bayesian Optimisation
0arXiv stat.ML
Swiss-Knife: A Framework for Reconfigurable Externalised Multi-Objective Alignment at Decode Time
0arXiv stat.ML
Inverse Design for Conditional Distribution Matching
0arXiv stat.ML