Methodology
How an idea earns
its place on your chart.
A backtest is a measurement, and a measurement is only worth acting on if it means what it says. This is the procedure that makes sure it does, and it is the same for our own research and for every idea screened here.
The one rule everything else follows from
Write down the test before you run it. Which gates, which thresholds, which markets, which era — all of it fixed in advance, in a ledger, with the trial counted.
This sounds bureaucratic and it is the whole game. A backtest drifts from the truth when a human looks at the result, adjusts something, and looks again, twenty times, and then reports the best one as though it were the only one. Fixing the test in advance keeps the number you read a measurement of edge, rather than a measurement of how many times you looked.
So every run is recorded, whatever its verdict, and the multiple-testing correction is applied against the true trial count. That is what makes a pass mean something.
To see what that correction does to a single backtest, put its Sharpe ratio and the number of variants behind it into the deflated Sharpe ratio calculator.
The gate stack
Each gate answers one question, and a candidate has to answer all of them. They run in this order deliberately: the cheap, decisive tests come first, so you get a clear answer before anyone spends a week on it.
The run on the right shows the shape of a screening pass and the checks each gate performs. The counts between gates are illustrative; where the run ends is the ledger’s own pass rate, 1 in 187. Select any finished gate to re-read its checks.
Candidates that pass, per 1,000
Select a gate above to see its checks again.
Causality
Could the signal have read a bar it could not have seen live?
Edge vs. noise
Is the average trade clearly above zero: over enough trades, not carried by a handful of sessions, and after counting every idea screened here?
Drift null
Does it beat random entries with the same trade count, hold and long/short mix?
Walk-forward
Does it hold on sessions the selection never saw?
Mirror test
Does the opposite trade, taken on the same bars, do clearly worse?
Cost
Does the edge clear real per-fill commission and slippage, with room to spare?
Era split
Did it also pay in the years before the sample it was judged on? Reported on survivors; it never decides a verdict.
The leak audit
Six leaks the audit
catches for you.
These are not exotic. Every one of them has been found in production research, and none of them shows up as an error — the backtest runs cleanly and reports a number that is about something else. Catching them is what turns a promising backtest into a strategy you can trust.
Wrong contract month
The backtest is reading a contract nobody was trading yet, or one that has already rolled off. Volume looks fine in aggregate and the prices are real — they are just not the prices anyone could have transacted at.
Roll-seam contamination
The join between two contract months creates a price jump the strategy trades. A seam error that is constant per contract is harmless; one that jitters is fatal, and the two look identical on a chart.
Calendar holes
Missing sessions, half days and holiday sessions silently change what a 'previous day' or 'first 30 minutes' means. A percentile computed across a sample containing holiday sessions is not the percentile you think it is.
Frozen features
An input that stops updating part way through the session — a session extreme that freezes at noon, a study that only recalculates on new bars. The strategy keeps trading a stale number and the backtest never complains.
A feed that arrives mid-sample
A composed signal whose external ingredient only exists for the second half of the history. The first half silently falls back to something else, so the strategy being measured is not the strategy being described.
The objective restates the parameter
Scoring in units the strategy itself defines. If the metric is average R and R is the stop, then tightening the stop improves the metric without improving anything real — and the optimizer will find that every single time.
What gets reported
Results are reported in dollars at a fixed risk budget, alongside the maximum drawdown, on every single summary. Never a return figure on its own, because a number with its drawdown beside it tells you what the strategy will actually be like to hold.
In a screen, costs are charged per fill at the real contract’s commission and a measured slippage, not estimated at the end. This matters more than it sounds: the same cost assumption can vary several-fold in risk-adjusted terms across instruments, so a strategy that clears costs on one contract can be underwater on another with an identical signal. The Trade Results record uses a flat two ticks a round trip per contract instead.
And where a market’s history allows, the result is split by era. An edge that held across different market conditions deserves more trust than one that held in a single stretch, so the report says which eras it worked in and how strongly.
Run the same procedure on your own idea. Describe it in plain English; the first verdict is free.
Or read the blog for longer notes on individual pieces of this.