A prediction market price is supposed to be a probability. When Polymarket or Kalshi prices a team at 87¢, the implicit claim is that teams in that spot win about 87% of the time. Do they? We can finally answer with live-captured, outcome-labeled data — the eventual winner joined onto every snapshot — instead of guessing.
The original report found low Brier scores for last-observed prices in a short MLB window. That is not sufficient by itself to establish calibration or pregame forecast quality. Last-observed in-play prices can contain near-terminal outcome information; the timestamp of the reference matters.
The setup
We pooled 41 distinct settled MLB game-instances across Polymarket and Kalshi (June 21–23, 2026; ~57,700 snapshots). For each game we took the closing (last-observed) price and compared it to the actual result.
- Closing-price Brier score: 0.136 (Polymarket) / 0.133 (Kalshi).
- A 10,000-iteration game-level bootstrap puts the 95% confidence interval at [0.090, 0.187] — entirely below the 0.250 you'd get from a coin-flip baseline.
- Favorites priced ~87% won ~86%. Underdogs priced ~21% won ~13%. Pearson correlation between closing price and outcome ≈ 0.70.
These are historical reported summary statistics, not a new October payload recount. Brier score measures more than calibration: it also reflects discrimination and outcome uncertainty. To assess calibration, inspect reliability bins and their counts, and predeclare the observation horizon. See the scikit-learn calibration guide.
The autocorrelation trap (this is the teaching moment)
Here's where most people go wrong. If you compute the Brier score over every row in the dataset, you get 0.239 — which looks like "barely better than a coin flip." You might conclude the market is sloppy. You'd be wrong.
Repeated observations within a game are dependent, so row-level independent confidence intervals can be misleading. But averaging Brier score over all snapshots and evaluating one last observation per game also answer different questions: they change time weighting and the information available at evaluation. The difference between 0.239 and ~0.13 is not explained by pseudoreplication alone, and neither estimate is automatically the “true” calibration number. For uncertainty, use game-level dependence; for forecast evaluation, choose a time-valid reference before inspecting outcomes.
Cross-venue: efficient, as expected
With both venues on one timeline, the mids correlate 0.9964, the median mid gap is 0¢, and a quoted cross-spread above an assumed 3c cost floor was reported in only 0.36% of snapshots — and even those average −0.52¢ once you pay the spread. These quote statistics do not establish simultaneous executable profit; we go deeper on that in our cross-venue arbitrage check.
Then we checked our own bot — and it doesn't beat the market
It would be easy to stop at "markets are calibrated and efficient" and quietly sell you a signal anyway. We won't, because our own trading bot proves the point against us.
Held to the same standard — the same Brier/calibration lens we walk through in our guide to measuring and fixing model calibration — our directional bot was down 18.6% on the full trade log at the time of writing. To be precise about where that loss lives: it is on live trades — backfill and shadow rows are excluded from our ledger, so the loss cannot be written off as a one-time backfill artifact. The more damning number is on the clean live cohort: the bot's claimed edge (edge_c) has zero monotonic relationship to realized P&L across edge bands — bigger "edge" does not mean more money, the per-trade result is roughly $0, and 2.15¢ of slippage eats into every fill. Those historical bot losses do not independently prove exchange efficiency or calibration. A bot can lose through model error, stale inputs, selection, costs or execution defects. We map out where sports markets are and aren't efficient in what our live results say about market efficiency.
One more honesty note, because it's the kind of thing that's easy to oversell: an internal log shows a tempting +17.5¢ "CLV" for the bot — but that figure is a measurement artifact (the "close" is captured ~43s after entry, biased toward the bot's own claim). The honest, settlement-time closing-line value was negative at the time of writing; the exchange-corrected aggregate now updates live on /clv — and we publish it whatever it says.
What we're not claiming
- N = 41 game-instances, 3 regular-season days, one sport (MLB), one market (moneyline), from a single capture window with a known Apr 22–Jun 14 gap. This is illustrative and early, not a seasonal or cross-sport verdict — which is exactly why we show the confidence interval instead of a lone point estimate.
- Three days do not establish stable calibration for other times or populations. The last-observation clock needs explicit checking before drawing a forecasting conclusion.
Reproduce it
The free, outcome-labeled matched-book sample is on Hugging Face and Zenodo (DOI 10.5281/zenodo.20816908). The winner is already joined to every row, so you can recompute the closing-price Brier in a few lines — and see how row/time weighting and the evaluation horizon change the estimand. The single-game sample alone cannot reproduce the full 41-game confidence interval.
For the full archive, see the data archives and how this data gets captured.