A prediction market price is supposed to be a probability. When Polymarket or Kalshi prices a team at 87¢, the implicit claim is that teams in that spot win about 87% of the time. Do they? We can finally answer with live-captured, outcome-labeled data — the eventual winner joined onto every snapshot — instead of guessing.
The honest answer, on the data we have: yes, closing prices are well-calibrated. But getting there involves a measurement trap that fools a lot of analyses, so the how matters as much as the what.
The setup
We pooled 41 distinct settled MLB game-instances across Polymarket and Kalshi (June 21–23, 2026; ~57,700 snapshots). For each game we took the closing (last-observed) price and compared it to the actual result.
- Closing-price Brier score: 0.136 (Polymarket) / 0.133 (Kalshi).
- A 10,000-iteration game-level bootstrap puts the 95% confidence interval at [0.090, 0.187] — entirely below the 0.250 you'd get from a coin-flip baseline.
- Favorites priced ~87% won ~86%. Underdogs priced ~21% won ~13%. Pearson correlation between closing price and outcome ≈ 0.70.
In plain terms: the market's closing price is a genuinely good probability estimate. No surprise to anyone who respects markets — but now it's measured on our data, with a confidence interval, not asserted.
The autocorrelation trap (this is the teaching moment)
Here's where most people go wrong. If you compute the Brier score over every row in the dataset, you get 0.239 — which looks like "barely better than a coin flip." You might conclude the market is sloppy. You'd be wrong.
The problem is pseudoreplication. A single game contributes hundreds of near-identical snapshots as its price drifts through the same probability bin. Those rows are not independent observations — they're the same game counted over and over. Pool them and the noisy mid-game prices swamp the sharp closing price, dragging the score toward 0.5.
Collapse to one closing observation per game and the true number appears: ~0.13. Same data, honest unit of analysis, completely different conclusion. This is the single most common way a prediction-market "study" misleads itself, and it's why we cite games, not rows, everywhere on this blog.
Cross-venue: efficient, as expected
With both venues on one timeline, the mids correlate 0.9964, the median mid gap is 0¢, and an executable lock above the 3¢ cost floor exists in only 0.36% of snapshots — and even those average −0.52¢ once you pay the spread. There is no free cross-venue money here; we go deeper on that in our cross-venue arbitrage check.
Then we checked our own bot — and it doesn't beat the market
It would be easy to stop at "markets are calibrated and efficient" and quietly sell you a signal anyway. We won't, because our own trading bot proves the point against us.
Held to the same standard — the same Brier/calibration lens we walk through in our guide to measuring and fixing model calibration — our directional bot is down 18.6% on the full trade log — though 95% of that loss is a sunk one-time March backfill, not live trading. The more damning number is on the clean live cohort: the bot's claimed edge (edge_c) has zero monotonic relationship to realized P&L across edge bands — bigger "edge" does not mean more money, the per-trade result is roughly $0, and 2.15¢ of slippage eats into every fill. That is exactly what an efficient, well-calibrated market predicts: if the price is already a good probability, a directional bot can't reliably beat it after costs. We map out where sports markets are and aren't efficient in what our live results say about market efficiency.
One more honesty note, because it's the kind of thing that's easy to oversell: an internal log shows a tempting +17.5¢ "CLV" for the bot — but that figure is a measurement artifact (the "close" is captured ~43s after entry, biased toward the bot's own claim). The honest, settlement-time closing-line value is negative: −6.7¢, positive only 42% of the time. We report the real one.
What we're not claiming
- N = 41 game-instances, 3 off-season days, one sport (MLB), one market (moneyline), from a single capture window with a known Apr 22–Jun 14 gap. This is illustrative and early, not a seasonal or cross-sport verdict — which is exactly why we show the confidence interval instead of a lone point estimate.
- Calibration this clean on three days does not guarantee it holds in playoffs, other sports, or chaotic in-game moments.
Reproduce it
The free, outcome-labeled matched-book sample is on Hugging Face and Zenodo (DOI 10.5281/zenodo.20816908). The winner is already joined to every row, so you can recompute the closing-price Brier in a few lines — and reproduce the autocorrelation trap by switching between per-row and per-game scoring.
For the full archive, see the data archives and how this data gets captured.