A model that says "Lakers win at 70%" can still be useless. The question isn't "were they right about who won." The question is: when it said 70%, did it actually win 70 of the next 100 similar games?
That is the calibration question. We do not have an exhaustive survey of sports models or vendors; a reliability study must define its particular population.
At ZenHodl, the public API serves 8 sports and the internal research fleet evaluates additional soccer, tennis, CS2, and LoL pipelines. Model coverage is not the same thing as live-money status; each trading lane can be live, shadowed, or dormant. Our calibration work is intended to make the live reference interpretable rather than turn win rate into a promise. Here's the full picture — what calibration is, why it is the quiet killer of sports models, and how to fix yours.
The 70% Thought Experiment
Imagine two models. Both predict NBA games. Both are 55% accurate (they pick the winner 55% of the time).
Model A says every game has a 55% chance for the favorite. Always. Same number.
Model B says some games are 80% favorites, some are 65%, some are 45%. It varies by matchup.
Both are equally accurate. But only one tells you anything useful. Model A has no discrimination within that population, but whether any selection has expected value still depends on the acquisition price. Model B might be tradeable, if its 80%-claimed games actually win 80% of the time.
That "if" is calibration. Being accurate (predicting the winner) is not the same as being calibrated (being right about how confident to be). A model can have perfect accuracy with terrible calibration, and vice versa. Neither accuracy nor calibration alone establishes profitable execution.
What Calibration Actually Means
Calibration is a property you can measure with one sentence:
When a model predicts probability p, the event happens with frequency p.
If you group all the games your model called "70% home win," you should see home teams winning 70% of the time in that group. Not 60%. Not 80%. Seventy.
Graph that property across all probability buckets, and you get a reliability diagram:
Perfectly calibrated model:
Predicted prob Actual freq
0.10 0.10
0.20 0.20
0.30 0.30
... ...
0.90 0.90
← diagonal line from (0,0) to (1,1)
Illustrative miscalibrated model:
Predicted prob Actual freq
0.10 0.19 ← overconfident at low probs
0.20 0.26
0.30 0.34
0.50 0.51 ← fine in the middle
0.70 0.62 ← overconfident at high probs
0.90 0.74 ← badly overconfident
The numbers above are illustrative reliability bins, not a measured industry result. A 90%-versus-74% discrepancy would require evidence from a stated bin and population.
How to Measure Calibration — ECE
Expected Calibration Error (ECE) is the standard metric. It's the weighted average of the gap between what the model predicted and what actually happened across probability buckets. Lower is better. Zero is perfect.
import numpy as np
def expected_calibration_error(predicted_probs, actuals, n_bins=10):
"""
predicted_probs: array of model probabilities, shape (n,)
actuals: array of 0/1 outcomes, shape (n,)
"""
bin_edges = np.linspace(0.0, 1.0, n_bins + 1)
ece = 0.0
for i in range(n_bins):
lo, hi = bin_edges[i], bin_edges[i + 1]
upper = (predicted_probs <= hi) if i == n_bins - 1 else (predicted_probs < hi)
mask = (predicted_probs >= lo) & upper
if mask.sum() == 0:
continue
bin_pred = predicted_probs[mask].mean()
bin_actual = actuals[mask].mean()
bin_weight = mask.sum() / len(predicted_probs)
ece += bin_weight * abs(bin_pred - bin_actual)
return ece
Report the binning, counts and reliability curve alongside ECE. A scalar under 3% or over 10% is not a universal quality threshold or per-prediction error bound.
For reference, here are the ECE values we track across our 11 sports (as of April 2026, measured on a seeded historical buffer of 84,960 predictions):
MLB: ECE 9.4% ← decent
NHL: ECE 4.1% ← excellent (low-scoring sport = stable prior)
CS2: ECE 4.9% ← good on training, drifts in live (more on this below)
ATP: ECE 9.6% ← moderate (surface-dependent)
LOL: ECE 1.0% ← looks great on paper, but...
The LoL number is interesting. It's gorgeous on historical data. It's terrible on our live trades (live ECE is often 24-28 percentage points). That gap is the heart of the next section.
Why Training ECE ≠ Live ECE (Adverse Selection)
Here's the trap: a model can be beautifully calibrated on random samples and catastrophically miscalibrated on the subset of samples you bet on.
Selecting trades based on model/market disagreement can change the evaluated population. Whether that selection harms calibration must be measured; it is not inevitable in every strategy.
In this illustrative strategy, the bot selects trades when its model disagrees with the market. It is paid only by the realized contract payoff; disagreement alone is not compensation or evidence of edge. But the market is not random. Polymarket prices are made by informed sharp players, bookmakers, and prop desks. When you think the market is wrong, it's usually because they saw something you didn't. Consciously or not, the bot is selecting for the hardest-to-predict cases.
Concretely: if your model trained to 4% ECE on random games, when you filter down to "games where my model disagrees with the market by 20+ cents," your live ECE on that subset could be 25%. The model isn't broken. The selection bias is eating it.
This is why we maintain a live recalibrator at ZenHodl. Every resolved trade updates a rolling isotonic regression per sport. The historical implementation could refit after a chosen number of outcomes. A 25–50-trade update interval does not certify detection of drift or successful correction.
You can watch this in action on zenhodl.net/live — every ORDER event has a fair_c value that's already been adjusted for live calibration drift.
How to Fix Miscalibration
Once you've measured the problem, there are three practical ways to fix it, in increasing order of flexibility (we walk through the same techniques with full worked code in why sports betting models lose money without calibration):
1. Platt Scaling
Fit a logistic mapping from model output to outcomes on separate calibration data. Logistic-regression fitting is numerical, not a general closed-form solution; evaluate performance on held-out predictions.
from sklearn.linear_model import LogisticRegression
def platt_calibrate(raw_probs, actuals):
X = raw_probs.reshape(-1, 1)
lr = LogisticRegression()
lr.fit(X, actuals)
return lambda p: lr.predict_proba(np.array(p).reshape(-1, 1))[:, 1]
Good when your miscalibration is roughly monotonic and your sample is small (< 500 resolved predictions).
2. Isotonic Regression
Isotonic regression fits a monotonic step function. Its flexibility can overfit a small calibration sample; choose and evaluate the method on separate data.
from sklearn.isotonic import IsotonicRegression
def isotonic_calibrate(raw_probs, actuals):
ir = IsotonicRegression(out_of_bounds="clip")
ir.fit(raw_probs, actuals)
return lambda p: ir.predict(np.asarray(p))
This is what we use in production. Flexible enough to handle S-curve miscalibration that Platt scaling can't capture.
3. Rolling Calibration Buffer
The above two are static fits. In live markets, you want calibration that updates as outcomes arrive. Keep a FIFO buffer of the last N predictions and actuals, and refit isotonic regression on each new datapoint:
from collections import deque
from sklearn.isotonic import IsotonicRegression
import numpy as np
class LiveRecalibrator:
def __init__(self, buffer_size=500, refit_every=25):
self.buffer = deque(maxlen=buffer_size)
self.refit_every = refit_every
self.ir = None
self._updates_since_refit = 0
def record(self, predicted_prob, outcome):
self.buffer.append((predicted_prob, outcome))
self._updates_since_refit += 1
if self._updates_since_refit >= self.refit_every and len(self.buffer) >= 50:
self._refit()
self._updates_since_refit = 0
def _refit(self):
probs, outcomes = zip(*self.buffer)
self.ir = IsotonicRegression(out_of_bounds="clip")
self.ir.fit(np.array(probs), np.array(outcomes))
def adjust(self, raw_prob):
if self.ir is None:
return raw_prob
return float(self.ir.predict(np.array([raw_prob]))[0])
This is the illustrative pattern described by the April article; it has not been verified as the October runtime for every sport. The illustrative buffer and update interval are implementation choices. Their suitability and drift-detection performance require evaluation; 25 outcomes may take more than a trading day.
Why This Matters More in Prediction Markets than Sportsbooks
Sportsbooks let you bet at fixed odds. An order can consume displayed liquidity or rest without a fill. A fill does not necessarily move the best quoted price. That has two consequences for calibration:
-
You can be right about the model and still lose. If your model says "70% home win" and Polymarket says "55% home win," you'd buy home. If you buy at 55c and a consistently defined pregame closing benchmark is 60c, gross bought-side CLV is +5c, before costs. The model’s 70c estimate is not the acquisition price. Near-terminal in-play closes require separate interpretation.
-
Your calibration affects your sizing, not just your side. A 55c purchase needs the true probability to exceed its total effective acquisition cost under the $1-payout model; there is no universal 58c threshold. If your model says 70c but it's actually calibrated to 61c, you're over-sizing by a huge multiple.
Miscalibration can distort sizing under either payout structure; the effect depends on the probability, odds, costs and allocation.
How We Use Calibration at ZenHodl
The April article described this historical implementation. The list below is retained as a dated report, not a newly verified October runtime:
- Every sport has a baseline ECE measured from seeded historical data
- Every resolved trade updates a rolling isotonic recalibrator for its sport
- Every live
EVALevent in our live trading feed shows the recalibratedfair_c, not the raw model output - Our results page reported an April snapshot of 501 trades, 56.5% WR and +$95.11. Those are historical article figures, not current P&L or a verified complete on-chain mirror
The calibration work is why we don't have to sell you a "69.8% historical win rate" we can't reproduce. Calibration alone does not explain a match between backtest and live returns or establish edge. Check current /results and /clv-evidence under their stated execution-provenance limits.
The One-Line Takeaway
If you're building a sports prediction model and you only track one thing besides accuracy, track ECE. A fixed 5% ECE threshold cannot diagnose the cause of an error or prescribe a profitable fix. Compare the traded and broader populations with bin counts, uncertainty and held-out evaluations.
Try It Live
The /live page previews a bounded set of recent recorded positions; it is not every evaluation, order or fill. /results shows a filtered ledger under its stated execution-provenance rules. The $49/month Starter plan provides model probability references through the documented API; inspect the dated benchmark and methodology for the model and sample you intend to use.
No backtest promises. Just the scoreboard, updated live.