Evaluating the Efficacy of Multi-Agent Systems: Can Patterns Be Automatically Selected?

In an investigation involving six multi-agent patterns across three benchmarks and models, the performance of agent teams—comprised of configurations like actor-critic or orchestrator-subordinate—was analyzed. The findings indicated that these teams often underperformed compared to single agents, both in terms of accuracy and cost, with some instances showing a cost increase of up to four times. While this might suggest a straightforward conclusion that multi-agent systems (MAS) are unnecessary, an exception emerged. The complexity of the task significantly influenced the potential for team success, raising two critical questions: Does the system's topology affect the outcomes? And if so, can we automate the selection of the appropriate pattern for specific tasks? Currently, the answer to the second question is negative, and the reasons behind this are particularly intriguing. The examples presented utilize the FEDOT.MAS multi-agent framework, with all experimental code and results available in a repository. While I measure the tool I developed, the conclusions drawn largely do not favor multi-agent systems, reflecting a degree of self-criticism rather than promotion.