Does a swarm beat one agent?

The same 20 historical claims from Darwin and Thompson were given to four set-ups with the same model, tools, instructions and budget. Each had to write a pre-registered test plan per claim that names one real public dataset. A script then checked every URL and DOI they wrote. Nothing here was typed by hand: every number is read from results.json.

Scoreboard

"Verified" means the dataset's landing page answered with HTTP 200 and mentions the dataset's name or publisher. "Fabricated" means a URL or DOI that does not exist (404, or a DOI the DOI system has never heard of). The rule was fixed before the runs: an arm beats another only with zero fabricated identifiers and at least 2 more verified datasets.

Verified datasets, out of 20

Wall-clock minutes

First worker launched to last worker finished. Agent-minutes (all calls added up) are in the table.

Experiment 2: the same four arms, web fetch required

Experiment 1's biggest caveat was that no arm used the web tools, so it measured feedback loops over memory. Experiment 2 reran the four arms on 20 new claims (no bird claims) with the rule that an agent must fetch the dataset's landing page before naming it. "Verified" here needs three things: the checker verifies the page, the run log shows the agent fetched that URL, and the page title the agent reports matches what the checker saw. Hard cost cap, no overshoot possible. Prediction and rules were written first: PREREG.md, section "Experiment 2".

Verified datasets, out of 20 (fetch required)

verifiedpage ok, no fetch evidenceunverifiedfabricated idno dataset / untouched

Claim by claim (experiment 1)

One row per claim, one column per arm. Click a cell to read the plan that arm wrote and what the verifier found.

verifiedURL ok, name not on pageunverifiedfabricated idno dataset / untouched

Inside the swarm

Queens, as measured

What the emperor sees

Centralized: king and emperor events

Decentralized: the shared board

Method and caveats

The four arms

arm	who decides	what it knows	feedback
single	one agent, one call, all 20 claims	all three portal briefs	none from us
isolated	20 workers in parallel, one claim each	the brief of its domain only	none: no board, no messages, no retries
centralized	emperor sets structure, king assigns claims to queens, reviews failures, retries, escalates to a curator queen	the queen's brief only	yes, through the king
decentralized	no planner: queens pull tickets from a shared board, post results, gaps and tips	the queen's brief plus the board	yes, through the board

Model , tools WebFetch and WebSearch only, no file access, run from an empty directory. Budget per arm: $3.00 at list price and 45 minutes. Queen briefs: briefs.json. Claims: claims.json.

Read before quoting a number

Twenty claims is small. A 2-claim margin is the pre-registered minimum difference, not a significance test. One run per arm is the primary result; replicates, if any, are listed in the table.
Verified is not "can decide the claim". It means the URL resolves to a page about that dataset. Whether the dataset fits the test is a separate judgement by an assessor from another provider, not done here.
Blocked is not fabricated. Some portals (GBIF's web pages, for one) refuse every non-browser client with HTTP 403. Those count as unverified, never as fabricated.
Not blind to birds. Class Aves is the project's held-out clade. Two claims here touch birds (darwin1859-ch01-10, thompson1917-ch15-01); the agents saw them, so they cannot be blind exam items. No organism data was downloaded by any arm.
The queens are coroutines in one Node process that only interact through the board file, not separate machines.

Pre-registration · Deviations and disclosures · nanopub drafts (unsigned, 3 claims + 3 evidence items) · metrics.json · emperor state

When does a swarm help? A sister result says: not always

A sister project, delphi-evolve (page: /evolve/), ran the same four-arm design on an enzyme benchmark at an equal 1,000-assay budget over 100 episodes, with zero false certificates in every arm. Their result: five specialists sharing evidence 78/100, the same five isolated 77/100, a single planner 78/100, and a single planner with its model prior switched off 96/100. Sharing beat isolation by one episode, tied the single planner, and the best arm was one agent with less machinery. That is the opposite of the result on this page (centralized 19 and 18 against single 13 and 12).

Both can be right, and the difference says when to expect a gain. Their specialists are deterministic code in one process: every member computes the same thing, so sharing adds nothing. Our members are language models with a context limit, and our per-claim table shows what the single agent actually lost. In experiment 2 all 8 of its losses were blank plans (honest nulls after 13 fetches for 20 claims in one context); it named no dataset that failed verification. In experiment 1 the split was 3 blanks and 4 named datasets that did not check out. So the single agent lost mostly by running out of room, and partly (experiment 1 only) by naming worse datasets. Collaboration paid where the bottleneck was attention and paid nothing where it was not.

Breadth of the win, checked per item rather than by totals: the centralized arm's verified-but-single-missed claims are 6 distinct claims in experiment 1 and 7 in experiment 2 (single won back 1), not one repeated item. Their benchmark is better controlled than ours on two counts, deterministic arms and every decision re-checked against full data; ours is closer to the agents anyone actually deploys. Both belong in the comparison.

Swarm thread of the DELPHI team, ScienceClaw hack, 2026-10-31. Built from files in swarm/compare/; rebuild with node swarm/compare/metrics.mjs && node swarm/compare/publish.mjs.