What did not replicate
Our strongest early effect collapsed from +11.3 points to +0.6 under randomized identities at ten seeds. Three-seed results are anecdotes. We publish ours as such.
- +11.3pp → +0.6pp
- public-reputation effect, 3-seed estimate vs. 10-seed replication
- 3/3 → 2/5
- paired seed wins of scalar reputation as seeds were added
What happened
In an early market with five tool-endowed agents, the information-regime ladder looked clean: oracle 97.5%, scalar reputation 94.3%, public granular 93.3%, private 89.3%, no signals 88.0%. Without signals, adverse selection appeared in textbook form, generalists underbidding specialists on tool tasks and failing them.
The robustness ablation with randomized agent identities did not reproduce the effect: +0.6 points instead of +11.3 on the tasks where it lived, with enormous variance across seeds.
It happened twice
In the model-tier market, scalar reputation beat the no-information baseline in 3 of 3 seeds with +18.6% mean welfare. At five seeds the mean fell to roughly zero and the paired wins to 2 of 5. The quality effect survived (accuracy 92% vs. 86%); the welfare effect did not. Pay-on-success contracts already cap the damage that reputation is supposed to prevent, and coarse signals ration tasks.
What we changed
Seed robustness became a primary outcome, reported as paired wins rather than point estimates. Confirmatory claims now run on fresh, pre-declared seeds against a criterion stated in advance; the broker result is the first that passed this bar. And we document where we walked into the garden of forking paths ourselves: the exploratory seeds were extended sequentially after results were visible, which is exactly why the preregistered confirmatory run existed.
Why it matters
LLM market experiments have huge seed variance, and the field mostly reports point estimates from a handful of runs. Treat those as anecdotes until replicated. Ours included.