Buying the Oracle
An intermediary that pays to test every agent and routes with the resulting competence matrix is the only information structure in our markets that robustly beats public reputation.
- +14.5%
- gross requester surplus vs. public scalar reputation
- 8/10
- paired preregistered seeds won (criterion: at least 7)
- 92% / 91%
- accuracy, broker vs. scalar. The gain is allocation, not error rate
- ~38 tasks
- market volume at which the calibration capex amortizes
The setup
Three model tiers (Haiku, Sonnet, Opus) bid in a reverse auction for math tasks across six calibrated difficulty levels. Costs are real: every inference call is deducted in dollars from the agent's wallet, including Opus's thinking costs, which scale with difficulty and are unknown to the model before it solves. Contracts are pay-on-success, and every answer is verified deterministically.
The intermediary condition adds two things. First, a paid calibration phase in which every agent is tested at every difficulty level, financed by the broker. Second, a private competence matrix built from that phase: who can do what at which level, not who is good on average.
What happened
Neither ingredient works alone. In a 2-by-2 design, exploration without granular bookkeeping burns its own capex, and granular bookkeeping without exploration fails at cold start. Only the combination covers its information cost.
The confirmatory run, ten fresh preregistered seeds, settles the headline: the broker beats scalar public reputation in 8 of 10 paired seeds, with mean gross surplus up 14.5%. Accuracy is statistically tied at 92% versus 91%, so the gain comes from allocation rather than error rate. Scalar reputation left 16 of 300 tasks unassigned; the broker left 2.
Honest shrinkage: the exploratory estimate was +22% over 8 seeds. Under preregistration it came down to +14.5%. We report the confirmatory number.
Why it matters
This is a classical result made concrete. Middlemen as experts (Biglaiser 1993) and certification intermediaries (Lizzeri 1999) are established theory; here the intermediary's expertise is a priced investment good. Calibration cost about $0.64 per market. At roughly 38 to 40 tasks the routing gain covers that capex; beyond it the broker is net positive, and it additionally compresses variance across seeds. It sells routing plus insurance against the unreliability of information-poor agent markets.
Limits
Three agents, one requester, math tasks, list prices. Bidding behavior is behavioral, not an equilibrium claim. And with a single requester, private and public information coincide: identifying the privacy moat of the broker requires the multi-requester experiment, which is the program's next main line.