The Aggregation Problem

Four Ways to Aggregate

Suppose three friends recommend restaurants for dinner. Alice says Italian, Bob says Japanese, Carol says Italian. How do you decide? There are at least four natural approaches:

• Majority vote. Italian wins 2–1. Simple, fast, democratic. But what if the question was "write a paragraph about Italian food"? You cannot vote on paragraphs.
• Best-of-K. You read all three recommendations and pick the one you find most compelling. This works for open-ended generation — you get the best individual response — but you ignore the potentially useful information in the other responses.
• Synthesis. You read all three recommendations and write your own, incorporating the best ideas from each. This is the most powerful approach, but it requires someone (or something) capable of synthesis. In multi-agent AI, this "someone" is the aggregator model.
• Debate. The three friends argue their cases, respond to each other's objections, and iterate until they converge on an answer. This is the approach used in multi-agent debate systems.

Four aggregation strategies: voting (for classification), selection (best individual), synthesis (new combined response), and debate (iterative convergence).

Why Voting Fails for Generation

Majority vote works beautifully for classification: each model outputs a class label, and you take the most common one. Condorcet's Jury Theorem guarantees that the majority is increasingly likely to be correct as the number of models grows (given independence and individual competence).

But most interesting tasks in LLM-based AI are not classification. They are generation: write an essay, summarize a document, answer a complex question in natural language. For generation, there is no natural "vote." Three models might produce three different but equally valid paragraphs. You cannot count which paragraph got the most votes, because no two responses are identical.

This is where synthesis and debate become essential. An aggregator model reads all three responses and produces a fourth response that is better than any individual — drawing on the strengths of each, correcting individual errors, and filling gaps that no single response covered. This is exactly what the Mixture of Agents architecture does: the aggregator in the final layer is performing synthesis.

The LLM-as-Judge Paradigm

A widely adopted approach to aggregation is LLM-as-Judge: use a (typically strong) language model to evaluate and rank the outputs of other models. This was formalized by Zheng et al. (2023) in the MT-Bench and Chatbot Arena framework. The judge model receives a prompt, the competing responses, and a rubric, and outputs a ranking or a pairwise preference.

LLM-as-Judge enables best-of-K selection without human evaluation: generate $K$ responses, have the judge rank them, and return the top-ranked response. It also enables tournament-style aggregation: run pairwise comparisons between responses, and use the tournament winner.

But LLM-as-Judge has known biases:

• Position bias. Judges tend to prefer the response presented first (or last, depending on the model). The order of presentation affects the ranking.
• Verbosity bias. Longer responses are rated higher, even when the additional length adds no information. Judges confuse length with quality.
• Self-preference. A model used as both generator and judge tends to prefer its own outputs. This creates a feedback loop that reinforces the model's biases rather than correcting them.

Condorcet's Jury Theorem: The Formal Guarantee

We state the theorem precisely and then examine its extensions to heterogeneous agents. Let $V_1, \ldots, V_K \in \{0, 1\}$ be independent binary votes with $P(V_k = 1) = p_k$. The homogeneous version assumes $p_k = p > 1/2$ for all $k$. The majority decision is $M = \mathbf{1}\{\sum_k V_k > K/2\}$.

Theorem (Heterogeneous Condorcet). If the votes are independent and $\bar{p} = \frac{1}{K}\sum_k p_k > 1/2$, then:

This generalization (via Hoeffding's inequality) shows that even with heterogeneous competence, the majority is reliable as long as the average competence exceeds $1/2$. Some agents can be below $1/2$ (worse than random) and the majority can still converge to the correct answer, as long as the strong agents compensate for the weak ones.

For weighted voting, where agent $k$ has weight $w_k$ and the decision is $\mathbf{1}\{\sum_k w_k V_k > \sum_k w_k / 2\}$, the optimal weights are $w_k^* = \log\frac{p_k}{1 - p_k}$ — the log-odds of each agent's competence. This gives more weight to more competent agents and effectively zero weight to agents at the chance level ($p_k = 1/2$). In practice, the competence levels $p_k$ are unknown and must be estimated, which introduces the additional challenge of learning the weights from data.

Arrow's Impossibility Theorem

Kenneth Arrow (1951) proved a fundamental impossibility result for aggregation of ranked preferences. Let $\mathcal{A}$ be a set of alternatives (e.g., candidate responses). Each of $K$ agents has a complete ranking of $\mathcal{A}$. A social welfare function $F$ maps $K$ individual rankings to a single group ranking. Arrow showed that no $F$ can simultaneously satisfy:

1. Unrestricted domain. $F$ works for any combination of individual rankings.
2. Pareto efficiency. If every agent prefers $a$ to $b$, then the group ranking has $a$ above $b$.
3. Independence of irrelevant alternatives (IIA). The group's relative ranking of $a$ and $b$ depends only on the individuals' relative rankings of $a$ and $b$, not on how they rank other alternatives.
4. Non-dictatorship. No single agent's ranking always determines the group ranking.

Theorem (Arrow, 1951). With $|\mathcal{A}| \geq 3$, no social welfare function satisfies all four conditions. Any function satisfying 1–3 is a dictatorship.

Arrow's impossibility: you must sacrifice at least one desirable property. Different aggregation rules sacrifice different properties.

Implications for Multi-Agent AI

Arrow's theorem has direct consequences for multi-agent AI systems. Consider $K$ LLMs that each produce a ranking of candidate responses. Arrow's theorem says there is no "perfect" way to aggregate these rankings into a single group ranking — every method violates at least one reasonable property. In practice, this manifests as specific failure modes:

• Majority cycles. Agent 1 prefers $a > b > c$. Agent 2 prefers $b > c > a$. Agent 3 prefers $c > a > b$. The majority prefers $a$ over $b$ (agents 1 and 3), $b$ over $c$ (agents 1 and 2), and $c$ over $a$ (agents 2 and 3). The majority preference is intransitive: $a > b > c > a$. There is no Condorcet winner.
• Manipulation. The Gibbard-Satterthwaite theorem (1973, 1975) extends Arrow's result: any non-dictatorial voting rule over three or more alternatives is susceptible to strategic manipulation. An agent can misreport its preferences to achieve a more favorable outcome. In multi-agent AI, this corresponds to the possibility that an agent might generate a strategically suboptimal response in order to influence the final aggregation in its favor.

Practical Aggregation Strategies

Given Arrow's impossibility, practical systems must choose which properties to sacrifice. Here are the main strategies:

The most interesting observation is that synthesis sidesteps Arrow's theorem entirely. Arrow's theorem applies to aggregation of rankings. When the aggregator generates a new response by reading and synthesizing all agent outputs, it is not selecting from a fixed set of alternatives or producing a ranking. It is producing a new alternative that may be better than any individual response. This is why the MoA architecture's synthesis-based aggregation is fundamentally more powerful than voting or selection: it operates outside the domain where Arrow's impossibility applies.

Multi-Agent Debate

Du et al. (2024) proposed multi-agent debate as an aggregation mechanism. The protocol is:

1. Each of $K$ agents generates an initial response to the prompt.
2. Each agent reads all other agents' responses and generates a revised response.
3. Repeat step 2 for $T$ rounds.
4. After $T$ rounds, take the majority answer (for factual questions) or the response of a designated aggregator.

Empirically, debate improves factual accuracy and reasoning quality. The mechanism is iterative error correction: in each round, agents that gave incorrect answers see the correct reasoning from other agents and can self-correct. Du et al. showed that on mathematical reasoning and factual QA benchmarks, 3 agents debating for 3 rounds outperformed a single model by a significant margin, and even outperformed more expensive chain-of-thought prompting.

Multi-agent debate: agents iteratively revise their answers after seeing others' responses. Convergence to the correct answer happens when the majority is initially correct.

When Agents Are Strategic

The analysis so far assumes that agents are honest: they generate their best response and report it truthfully. But in multi-agent systems, agents may have incentives to be strategic. If an agent knows how its output will be aggregated, it may modify its output to influence the final result. This is precisely the setting studied in mechanism design.

Consider a concrete scenario. A multi-agent system uses best-of-K selection via LLM-as-Judge. Agent $A$ knows that the judge has a verbosity bias (longer responses are rated higher). Agent $A$ can improve its chances of being selected by padding its response with extra detail, even if that detail is irrelevant or redundant. The result: the aggregation mechanism incentivizes a specific kind of suboptimal behavior, and the equilibrium output is worse than it would be under honest reporting.

Mechanism Design for LLM Aggregation

Mechanism design asks: can we design the aggregation rule so that honest reporting is the optimal strategy for each agent? The Vickrey-Clarke-Groves (VCG) mechanism achieves this for certain utility structures, but it requires monetary transfers — not directly applicable to LLM systems.

For multi-agent LLM systems, the relevant mechanism design question is: can we design an aggregation protocol under which each agent's best strategy is to generate its highest-quality response?

One approach is the peer prediction framework. Instead of evaluating each agent's response against the true answer (which may not be available), evaluate it against the other agents' responses. Specifically, score agent $k$'s response $r_k$ using:

The first term rewards agreement with other agents (encouraging accuracy). The second term rewards information not present in other responses (encouraging diversity). The balance parameter $\lambda$ controls the tradeoff. Under certain conditions on the similarity metric and the agents' information structure, honest reporting is a Nash equilibrium of this scoring rule.

Nash Equilibria in Multi-Agent Debate

Multi-agent debate introduces a dynamic game. At each round, each agent chooses a response (action). The payoff depends on the final consensus (if the debate protocol rewards agents whose position becomes the consensus) or on a judge's evaluation. We can analyze this as a repeated game.

Define the debate game $\Gamma = (K, \mathcal{A}, u, T)$ where $K$ is the number of agents, $\mathcal{A}$ is the action space (possible responses), $u_k : \mathcal{A}^K \to \mathbb{R}$ is agent $k$'s utility, and $T$ is the number of rounds. At each round $t$, each agent observes the previous round's actions $a^{(t-1)} = (a_1^{(t-1)}, \ldots, a_K^{(t-1)})$ and chooses $a_k^{(t)}$.

A truthful equilibrium is a strategy profile where each agent's best response at every round is to report its genuine belief. The key question: under what conditions on $u$ does a truthful equilibrium exist?

Proposition. If the utility function rewards agents for being in the majority consensus and the consensus is more likely to be correct than incorrect (i.e., the Condorcet condition holds), then reporting the truth is a Nash equilibrium of the one-shot game. In the repeated game, truth-telling is a subgame-perfect equilibrium if agents use trigger strategies (punish observed dishonesty by reverting to a less cooperative equilibrium).

The proof follows from the fact that, under the Condorcet condition, an agent that reports truthfully is more likely to end up in the winning majority than one that deviates. The utility from being in the majority exceeds the utility from any alternative strategy. However, this breaks down when:

• The Condorcet condition fails (agents are more likely wrong than right).
• Agents can coordinate to form a bloc (collusion).
• The debate protocol has asymmetric information (some agents know more than others).

Convergence of Iterative Debate

Does multi-agent debate converge? And if so, to what? Model the debate as a dynamical system. Let $\mu^{(t)} \in \Delta(\mathcal{A})$ be the empirical distribution of agents' responses at round $t$. The debate dynamics can be modeled as:

where $F$ represents the aggregate effect of all agents updating their responses based on the current distribution. If $F$ is a contraction mapping (i.e., $\|F(\mu) - F(\nu)\| \leq \gamma \|\mu - \nu\|$ for some $\gamma < 1$), then by the Banach fixed-point theorem, the dynamics converge to a unique fixed point $\mu^*$.

When is $F$ a contraction? If each agent updates by moving toward the consensus (e.g., adopting the majority position with high probability), then $F$ is contractive. The contraction rate $\gamma$ depends on how "stubborn" agents are: agents that give high weight to their own prior opinion produce a $\gamma$ close to 1 (slow convergence); agents that fully adopt the majority produce $\gamma$ close to 0 (fast convergence, but susceptible to cascade effects).

The fixed point $\mu^*$ is not necessarily the correct answer. It is the consensus — the point where agents stop changing their responses. If the initial majority is correct (Condorcet condition), the consensus is likely correct. If the initial majority is wrong (e.g., all models share the same hallucination), the consensus reinforces the error. This is the multi-agent analogue of groupthink: convergence to a wrong answer that appears unanimous.

Debate converges under contraction conditions, but the fixed point may be correct or incorrect depending on the initial distribution of correct answers.

The Price of Anarchy

The price of anarchy (Koutsoupias and Papadimitriou, 1999) measures how much worse the equilibrium outcome is compared to the social optimum. For multi-agent debate, define:

where quality is the accuracy or relevance of the final output. A PoA of 1 means the equilibrium is optimal; a PoA of 0 means the equilibrium is arbitrarily bad.

For simple majority voting with $K$ honest agents, the PoA is 1 (the equilibrium is truth-telling, which is optimal). For debate with strategic agents, the PoA depends on the reward structure. If agents are rewarded for accuracy (e.g., their response matches the ground truth), the PoA is close to 1. If agents are rewarded for agreement (e.g., being in the majority regardless of correctness), the PoA can be much less than 1 — the equilibrium may be consensus on a wrong answer.

This analysis has practical implications for multi-agent system design: the reward signal for the aggregation mechanism determines whether the system converges to truth or to groupthink. Rewarding accuracy incentivizes honest reporting; rewarding agreement incentivizes conformity. Real systems should reward accuracy whenever ground truth is available, and fall back on diversity-encouraging mechanisms (like the peer prediction score above) when it is not.