Most published trading results share a quiet assumption: that the ledger behind them is accurate. Nobody audits their own fill prices against the exchange's records — least of all when the numbers already look acceptable.
We just spent five days doing exactly that to our own Polymarket trading bots, and the headline is uncomfortable: 74% of our fast-path MLB entry prices didn't match the exchange's own record of what we paid. The skew averaged +1.78 cents in our favor — our published results looked slightly better than reality.
This post is the full engineering log: the bug, the reconciliation, the corrected (worse) public numbers, and the execution-layer hardening that came out of it. If you run — or are building — a prediction-market trading bot, every one of these failure modes is waiting for you too.
The bug: recording the limit, not the fill
When you submit an order to Polymarket's CLOB, the order-placement response tells you the order was accepted. What it does not carry is an average execution price — there is no avgPrice field in the placement response.
Our fast fill path did what most bot code quietly does: it recorded the submitted limit price as the entry price. If we offered 48c and got filled at 46c, the ledger said 48c. If partial fills walked the book, the ledger didn't know.
The exchange truth was recoverable the whole time — the response's makingAmount and takingAmount fields encode the actual executed price and size (for a BUY: price = making/taking, size = taking). We just weren't reading them.
Measured against exchange records:
| Cohort | Entries mismatched | Mean skew |
|---|---|---|
| MLB fast-path (pre-August) | 74% | +1.78c in our favor |
The same reconciliation surfaced an April sizing bug whose rows carried garbage sizes entirely — roughly $54 of losses that our own dashboard had never shown.
Fixing history without rewriting it
Our trade ledger is append-only by doctrine — rewriting historical rows is how trading operations quietly launder their past. So the fix has two layers:
Live, going forward: every fill price now derives from the exchange's own amount fields, with an order-status confirmation fallback. Every row carries a fill_price_source provenance tag. A fill that can't be verified is labeled unverified rather than silently trusted — the failure mode where a fallback pretends to be a confirmation is structurally dead.
Historical rows: exchange truth lives in a public reconciliation sidecar, keyed to each original trade — matched rows carry the exchange's entry price, size, P&L, and closing-line value next to what we originally recorded. You can download the raw sidecar and check our math.
Publishing the worse number
Corrected fill prices flow straight into closing line value, and our public CLV pages now inject exchange truth everywhere it exists. The corrected aggregate:
- 47.4% of fills beat the close (was 47.6%)
- Mean settlement CLV: −1.40c (was −2.0c at our July retraction, now measured on true fills)
Still negative. Still published. Our operating rule is simple: when a fix makes a number worse, ship the worse number. A trading operation that only corrects errors in the flattering direction isn't correcting errors — it's curating them. The full methodology and every prior retraction live on /clv-evidence.
The frozen-quote problem
The same audit window closed a subtler execution bug. Our quote-staleness gates — the checks that should stop the bot from trading on a dead order book — were re-stamping local receipt time on every cached quote copy. A book that froze mid-game would still look "2 seconds fresh" forever. Every staleness gate we had could literally never fire.
The fix plumbed true per-token source timestamps through both price trackers, and after 12 days of log-only evidence we flipped enforcement on: entries and pre-submit requotes are now blocked on any traded book whose source feed has gone silent.
One honest nuance that log-only period taught us: at any moment, hundreds of tracked books are "source-quiet" for more than 30 seconds — and most are legitimately quiet pregame markets where nothing is trading. Raw quote age can't be the enforcement key across the board; the gate keys on books being actively traded. If we'd enforced on day one instead of logging for two weeks, we'd have blocked half the fleet for no reason.
One order builder for live, shadow, and probe
Three smaller fixes from the same pass, each closing a way a bot lies to itself:
-
Shared order construction. Live orders, shadow simulations, and probe orders now price through one shared builder with one executable-price cap. Before this, a shadow fill could simulate a different order than the live path would have sent — which quietly flatters shadow results, and shadow results are what we use to decide whether a sport goes live. Shadow evidence is only evidence if it simulates the trade you'd actually place.
-
Post-poll price refresh. All five bot plugins refreshed prices before polling game state, so decisions could ride prices a full poll cycle stale. The refresh now happens after the poll, immediately before the decision.
-
Order-failure cooldown. A failing order path now backs off with escalation instead of retrying into the same wall — the retry-storm class where one structural failure becomes a hundred log lines and a rate-limit risk.
Model governance, same week
The execution audit ran alongside a fleet-wide model audit — external claims, each adversarially verified before we acted. The confirmed ones each got a structural fix:
- Soccer live trading is frozen. The soccer XGBoost artifact had silently vanished from the server in April, silently degrading the bot to a Poisson fallback that measures worse than its own entry prices (n=87, and the confidence interval excludes zero — that's not noise). We retrained a clean challenger; it failed its promotion gates; the freeze stands. A daily artifact-presence monitor now pages if any of 25 model files disappears again.
- LoL reverted to its v1 analytical model — twice. Head-to-head on 110 fire-anchored matches, the fancier v2 override lost (Brier 0.2077 vs 0.1692). We found and fixed v2's starved form and head-to-head inputs, re-ran the whole evaluation, and v2 still lost. Two independent reads, same verdict, revert stands.
- A recalibrator guard, with a disclosure. WNBA's live recalibration layer was saturating high-confidence predictions to 0.995 off a two-sample isotonic tail — and those saturated probabilities were served on our public API. Sparse-tail corrections are now blocked fleet-wide. Subscribers saw those numbers, so this is stated here plainly rather than buried.
- NFL's situational features were dead — and nothing noticed for months. A fleet-wide feature-importance audit found down, distance, and field position had trained as constant zero; the model had learned to ignore them. The retrain fixed the supply contract — identical game states with different situations now move the prediction by up to 18.5 points where the old model moved zero. NFL stays shadow-only for the season start anyway: our go-live gates run on accumulated evidence, and the gate does not care that the model got better.
- A monthly feature-drift tripwire now compares every model's feature-contribution profile against a stored baseline and pages when a feature dies or wakes — the exact class of silent breakage this audit kept finding.
Every skip now leaves a record
The final batch closed the observability gap on the other side of the ledger. Bots log what they do; almost nothing logged what they declined. Eleven new audit points now record every gate rejection — edge floor, price bounds, spread cap, direction filters — with full price context, rate-limited so the log stays readable. And every trade record, live or shadow, now carries the order book's spread and both-side depth at signal time, because our microstructure review surfaced a hypothesis we couldn't test historically: fills into deep resting offers looked meaningfully worse than fills into thin ones. Forward evidence will settle it.
Why publish any of this
Because the alternative is the industry default, and the industry default is why nobody trusts published trading results.
Every number in this post is checkable: the reconciliation sidecar is public, the corrected aggregates live on /clv-evidence, the version-by-version detail is on the changelog, and our negative results — retractions included — are archived on /research. The same infrastructure that caught these bugs is what produces our data products: score-synced order book archives captured live, with provenance labels on every measurement, because we learned the hard way what happens without them. If that's the kind of data discipline you want under your own models, start with the historical data overview — the free samples carry the same schema honesty as the paid archives.
The bots are roughly breakeven and we say so. The moat was never a secret edge — it's a ledger you can audit.