Backtesting honestly

Every strategy someone tries to sell you comes with a backtest, and the backtest always looks good. That is not a coincidence, and it is not usually fraud. It is a predictable consequence of how testing works when nobody counts the attempts.

The problem with trying things until one works

Suppose you test a moving-average crossover. It loses. You try different lengths, then add a filter, then restrict it to certain hours, then skip Fridays. Eventually one combination produces a beautiful equity curve.

You have not found an edge. You have searched a large space of variants against one fixed history and picked whichever fit it best — and some variant always fits, because random data contains patterns too.

This is measurable, not folklore. Bailey, Borwein, López de Prado and Zhu showed in the Notices of the American Mathematical Society that a high simulated performance is easy to achieve after testing a relatively small number of strategy configurations, and that the more configurations you try, the more likely it is that the best backtest is overfit — fitted to noise that will not repeat. Their sharper point is about disclosure: because almost nobody reports how many variants they tried, you cannot judge how overfit a result is. A backtest presented without that number is not evidence.

Ask any signal seller or course vendor how many versions they tested before the one in the screenshot. The question is usually unwelcome, which is itself informative.

Test so the number survives contact with the future

Decide the rules before you look. Write the entry, the exit, the stop and the size down first. A rule you added because it improved the curve is a rule fitted to that history.

Hold data back, and mean it. Develop on one period, then test once on a period you have never looked at. Once. If you tune after seeing the out-of-sample result, it is no longer out of sample — it has become part of your search, and you are back where you started.

Count and record your attempts. Keep a log of every variant you tried. Fifty attempts to find one good result is a strong signal the result is noise; three attempts is a weaker one. The number costs nothing to keep and is the single most honest statistic in your whole test.

Include the costs that will actually apply. Spread on every trade, slippage on stops, and swap on positions held overnight. A strategy averaging four pips per trade is destroyed by a two-pip spread it did not model — see reading a currency pair.

Prefer fewer parameters. Each one you add is another dimension to search, and another way to fit history. A rule with two parameters that works acceptably beats one with nine that works beautifully in the past.

Read the equity curve, not the total

Total return is the least useful number a backtest produces. What matters is whether you could have kept trading it:

• Maximum drawdown — the deepest peak-to-trough fall. If that number is 40%, the recovery required is 66.7%, and you should ask honestly whether you would have kept going. See drawdown math.
• The longest losing streak — not how much, but how long. Most people abandon a system during its worst stretch, which turns a profitable strategy into a realised loss.
• How the return was earned — a curve carried by two enormous winners out of three hundred trades is a different proposition from one built on consistent small edges, even at identical totals.
• Number of trades — thirty is an anecdote. A result from a small sample tells you about that sample.

Why the regulator makes firms say it

FCA rules on financial promotions require past performance to be presented with prominent warnings that it does not indicate future results, alongside the period covered and its source. That requirement exists because historical performance reliably persuades people, and reliably fails to predict.

The rule binds firms. The reasoning applies just as much to the backtest you ran yourself last night — with the added difficulty that you know exactly how much you want it to be true.

What a good backtest actually buys you

Not a forecast. It buys you a rejection: evidence that an idea does not work, cheaply, before it costs real money. Most tested ideas fail, and that is the point — the value is in the ones you decline to trade.

For the surviving handful, a backtest gives you a rough expectation of drawdown and streak length, so you know what normal looks like and can tell it apart from a broken system. That is worth having, and it is all it is worth.

Next: building a system worth testing in the first place.