Does the system work the way the backtest promised? How we measure the strategy's health live
· Strategy
Anyone who trades a backtested system eventually asks the same question. A third losing trade in a row: is it ordinary bad luck the backtest allowed for, or a sign that the market has changed and the strategy has stopped working? Looking at single trades settles nothing. A backtest does not promise that every entry makes money, only a certain distribution of results - losing streaks included.
So we built a module that asks this question every day and answers with numbers. This article describes how it works and what it showed on its first day.
1. Every trade against its own backtest
The starting point is every trade of every scenario in the strategy's backtest, on three runs with shifted entry times. A trade's result is the price move in the position's favour, without leverage. That way accounts with different leverage and position sizes compare with the backtest and with each other.
Each live trade gets a percentile: the share of the same scenario's backtest trades that did worse. If the system works as in the backtest, the percentiles spread evenly and average about 50%. A single trade at the 5th percentile means nothing - one in twenty backtest trades was that weak too. Only an average over many trades clearly below 50% says something has changed. How clearly depends on the number of trades: with 4, chance easily pushes the average to 35%; with 40, it does not.
2. An open position: where it should be after this many hours
Closed trades are hindsight. For open positions we compare the current result with where the backtest trades were at the same age - after 1, 3, 6, 12, 24 hours and longer - for all of them and separately for those that ended in profit.
An example from the first day: a Breakout Down SHORT on BTC, open on two accounts, was 2.5% down after 4.6 hours. In the backtest this scenario's median at that age is +0.8%, and no trade was that weak at that point - percentile 0%. This is not a signal to close: we had already checked that leaving positions early, for example on an opposite scenario's signal, clearly lowers the result. But it says this position is behaving unlike anything the strategy has seen in its history.
3. The market or the execution?
The first report showed a clear signal. 19 live signals since 11 September averaged the 33rd percentile of the backtest:
| Measure | Live | Backtest of the same scenarios |
|---|---|---|
| Mean percentile | 33% | 50% (by definition) |
| Winning trades | 63% | 82% |
| Mean move per trade | +0.08% | +3.1% |
A gap like this can have two causes. Either the market of these weeks was bad for these scenarios, or the live trader does something differently from the simulation: enters later, exits earlier, pays slippage. To tell them apart we check the backtest over the same period. It was similarly weak - the 39th percentile on BTC, the 34th on SOL. Most of the gap is therefore the market: September and October were hard, especially for the Puell Hot Trend LONG scenario. At the same time the backtest over the whole last 180 days, pooled over all scenarios, is normal (the 47th percentile) - this is not a slow loss of edge but a weak period.
A smaller part of the gap is execution. On the same entries Puell Hot Trend LONG made +0.3% live on 30 September against +2.4% in the backtest, and on 15 October it exited flat where the backtest made +2.4%. Pairs like these point to specific trades to compare step by step with the simulation.
4. One number - and why it needs a second one
For a quick view of the whole there is system quality on a 0-100 scale: twice the mean percentile, so 100 means "as the backtest or better" and 50 means the typical trade lands at the 25th percentile. With few live trades the score is pulled towards the backtest of recent months, as if that were ten extra trades. That way the first two trades do not overturn the assessment, and with a few dozen they decide on their own.
The first version had one flaw: a score of 76 lit up red. The number said "not bad", the colour said "alarm". Both were true, because they answered different questions: how much worse, and how sure we are of it. We separated them: the wording depends on the number alone (90-100 as the backtest, 75-90 weaker, under 75 much weaker), and certainty is given separately, as a percentage. The colour appears only from 95% certainty. The first reading: 76 - weaker than the backtest, 99% certain.
5. Which scenarios actually carry the result
The same material gives a ranking. A scenario's contribution is its mean move per trade times its trades per year - per year, because BTC's history in the ETF era is longer than SOL's. At the top are two very different kinds: Exchange Inflow Fund Outflow SHORT on SOL (13 trades a year at +2.9%) and Weekly Trend Dip LONG on BTC (27 trades at +1.4%). Right behind them are rare but large entries: Leverage Flush SHORT on SOL trades under three times a year, but averages 11%. At the bottom are scenarios that hardly trade in the ETF era or once did harm - candidates for review.
The quarterly results also show slow trends that a single test does not. Weekly Trend Dip, the most active scenario, averaged 1.7-4.6% per trade in 2024, 0.3-1.1% in 2025 and −0.7% to +1.5% in 2026. It still makes money, but less per trade - worth watching before it becomes a problem.
6. The backtest and real trading over time
The last element is a chart: the monthly sum of price moves in the backtest and live, from the first live trade. The key caveat: the backtest simulates today's config, and that config changed throughout the year.
| 2026 | Backtest | Live |
|---|---|---|
| February-July (monthly sums) | +11% to +62% | −29% to +10% |
| August | +36% | +48% |
| September | +28% | +30% |
| October | −11% (to 21.10) | −31% |
Until July live trading was much weaker than the backtest - but a different, earlier config was running then, while the backtest runs everything on today's. From August, on settings close to the current ones, real trading kept pace with the backtest, and in August it beat it. October is the first month in which both are negative, with a clearly worse live result - exactly the period the quality score marked as weaker.
Limits
- Most scenarios trade a few times a year. For them conclusions stay uncertain for months, and the module says so instead of lighting up red after two weak trades.
- The reference is the backtest of a specific config. After every larger change to the strategy a new reference has to be computed.
- A move without leverage compares the quality of entries and exits, not position size - the account result judges that separately.
Summary
- A single trade says nothing about the system. Its percentile in the backtest's distribution, averaged over many trades, does.
- First reading: system quality 76/100, weaker than the backtest, 99% certain.
- The backtest of the same weeks was similarly weak, so it is mainly the market; a smaller part of the gap comes down to specific live exits.
- The ranking and quarterly results show which scenarios carry the result and which are slowly losing their edge.
- From August, on a config close to today's, live trading kept pace with the backtest.
The main lesson is simple: whether a system has stopped working is decided not by the last trade or the last week, but by whether results drift clearly and lastingly away from what the backtest considered normal.