Does the Live Trader Do What the Backtest Says? Our Daily Prod-vs-Backtest Check
· Framework
Everything we decide about our BTC strategy rests on backtests: which scenarios stay, how big a position is, where the stop sits. A backtest is only worth something if the live trader does the same thing on the same day. So we started checking exactly that — every day, automatically, step by step.
Within the first week the check found two real differences between live trading and the backtest. Neither was a bug in the strategy, and neither would have shown up in a profit-and-loss chart.
1. How the daily check works
Every night, after midnight UTC, the backtest dataset gets the day that has just finished: one-minute Binance candles and every indicator computed for each 10-minute step by the trader's own code. Right after that a report compares the day three ways:
- Indicators, step by step. The live trader logs about 40 values every cycle — trends, pivots, breakouts, distance from the last peak and trough, support and resistance slopes, trend momentum, Bollinger %B, CCI, volume oscillator, ATR, price. Each is compared with what the backtest saw at the same 10-minute step: enums as the share of steps that agree, numbers within a tolerance.
- On-chain data. Every on-chain value the trader logs (SOPR, MVRV, exchange netflow, funding rate and others) against the value the backtest reads for that day.
- Trades. For every account trading live, its trades that day against a backtest of its current configuration (with 30 days of warm-up, so positions can already be open). Trades are paired by side, scenario and opening time.
2. Finding one: the live trader looked at the market at the wrong minute
The first report showed something odd: the price momentum indicator — whether the smoothed price rose over the last 10 minutes — agreed with the backtest on only 48% of the steps. That is a coin flip. Several scenarios use it as an entry condition.
The logic was identical on both sides. The difference was when it ran. The backtest computes every step at minute 5 of the 10-minute window. The live trader started a cycle 600 seconds after the previous one, so its moment drifted through the window — on one day it looked at the market around minute 1, on another around minute 5, and every restart reset it. For a slow indicator that does not matter. For a ten-minute price change it decides everything: the typical change over ten minutes was about 57 USD, while sampling a few minutes apart moved the price by 24 USD on average and up to 200 USD.
One day mixed both kinds of cycles, which made the effect easy to measure:
| Indicator | Cycles at minute 5–7 | Cycles at other minutes |
|---|---|---|
| Hourly trend momentum | 98% | 25% |
| Hourly CCI | 100% | 51% |
| Hourly Bollinger %B | 92% | 45% |
| Price momentum | 92% | 77% |
| Market price | 98% | 69% |
| ATR | 100% | 88% |
The fix was one change to the scheduler: live cycles now start on the clock at minute 5 of every 10-minute window (00:05, 00:15, …) — exactly when the backtest computes its steps. The strategy itself did not change; the live trader now looks at the market at the moment the backtest assumes.
3. Finding two: "fresher" on-chain data is different data
Most on-chain values agreed with the backtest on 94–100% of the steps. One did not agree on a single step: exchange netflow.
The data provider fills in today's record during the day. The live trader reads the latest record available, so in the morning it sees a netflow of, say, 66 BTC, which grows through the day. The backtest only has final daily values, so it uses yesterday's full total — 5,692 BTC on the same day. These are not the same number seen earlier; they are different quantities. A threshold tuned on daily totals means something else when applied to a partial sum at 9 a.m.
Is the faster data better? Nobody can say, because it has never been tested: history keeps only final daily values, and using today's final value in a backtest would mean knowing the future. So we kept the live trader as it is and started recording, every 10 minutes, exactly what it sees. In a few months these snapshots will let us backtest the faster data honestly — and decide with numbers instead of intuition.
4. Measuring the right thing
A comparison is only useful if it compares like with like. Two early "mismatches" were mistakes of the report, not of the trader:
- Breakouts. The trader logs a breakout after its filters (a breakout counts only near the last pivot). The dataset stores the raw crossing. Compared raw, the weekly breakout agreed on 0% of the steps; with the same filters applied, on 100%.
- Distance from the last peak and trough. It is measured in percent of the last swing, and an hourly swing can be only about 100 USD wide. A price difference of 30–50 USD is then several percentage points — noise, not disagreement. The report now uses a tolerance per timeframe.
5. Trades: the same decision, a little later
Paired trades are the most telling part. For example, two accounts with the same configuration opened a long from the Puell scenario two and a half hours after the backtest did — the same decision, at a slightly different moment and price. Unpaired trades so far have simple explanations: a position opened by hand, or a trade that began before the backtest's 30-day window.
Summary
A backtest answers "what would the strategy have done". It does not answer "what does our live system actually do". The daily comparison closes that gap:
- a scheduling detail made one entry condition random in live trading — fixed by starting cycles on the clock;
- a data-freshness difference changes one on-chain input — now recorded, to be tested rather than guessed;
- every day from now on, any new difference between live trading and the backtest shows up the next morning, before it can cost money.
The rule behind it is the same as in our testing: trust a result only after checking it in the conditions it will actually run in.