.png)
May 8, 2025
Large language models (LLMs) impress with their complex reasoning abilities. But how well can they cooperate in multi-agent systems (MAS) when they – similar to natural swarms – have to operate under strict conditions? This question is at the center of current research exploring the potential of LLMs for decentralized coordination and swarm intelligence.
Existing benchmarks often inadequately represent the challenges of decentralized coordination arising from incomplete spatio-temporal information. A new benchmark system called SwarmBench aims to close this gap and systematically evaluate the swarm intelligence of LLMs. The special feature: The LLMs act as decentralized agents in a configurable 2D grid environment and must rely primarily on local sensory input (k x k field of view) and local communication.
SwarmBench comprises five fundamental MAS coordination tasks that the agents must master under these restrictive conditions. The tasks simulate various scenarios in which cooperation and coordination are crucial. To measure the performance of the LLMs, new metrics for the effectiveness of coordination and the analysis of emergent group dynamics have been developed. These metrics allow for a differentiated assessment of the LLMs' abilities across the various tasks.
Initial tests with leading LLMs in a zero-shot setting – meaning without prior training on the specific tasks – show significant performance differences. While some LLMs already show initial approaches to coordination, the results also reveal difficulties in robust planning and strategy formation under uncertainty in these decentralized scenarios. The limitation to local information, in particular, presents a major challenge.
SwarmBench is provided as an open and extensible toolkit. It is based on a customizable and scalable physical system with defined mechanical properties and includes environments, prompts, evaluation scripts, and the generated experimental datasets. This is intended to promote reproducible research in the field of LLM-based MAS coordination and the theoretical foundations of Embodied MAS.
Evaluating LLMs under swarm-like conditions is crucial to fully realizing their potential for future decentralized systems. SwarmBench provides researchers with a valuable tool to explore the limits of current LLMs and to advance the development of more robust and effective algorithms for decentralized coordination.
Bibliography: - Ruan, K., Huang, M., Wen, J.-R., & Sun, H. (2025). Benchmarking LLMs' Swarm intelligence. *arXiv preprint arXiv:2505.04364*. - https://huggingface.co/papers/2505.04364 - https://huggingface.co/papers - https://arxiv.org/abs/2502.09933 - https://arxiv.org/abs/2410.07166 - https://proceedings.neurips.cc/paper_files/paper/2024/file/b631da756d1573c24c9ba9c702fde5a9-Paper-Datasets_and_Benchmarks_Track.pdf - https://openreview.net/pdf?id=L0oSfTroNE - https://www.researchgate.net/publication/388094928_Dynamic_Intelligence_Assessment_Benchmarking_LLMs_on_the_Road_to_AGI_with_a_Focus_on_Model_Confidence - https://papers.cool/arxiv/2501.07572 - https://github.com/zhangxjohn/LLM-Agent-Benchmark-List - https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5239555