Backtesting, optimisation and overfitting

A backtest runs a set of trading rules over past prices and records the trades they would have made. It is the usual evidence offered for an automated system, and it is easy to produce one that looks excellent and means nothing. This lesson explains how a backtest works, what optimisation does to it, and the two standard checks: out-of-sample testing and walk-forward testing.

What a backtest is

The test steps through historical prices one bar or one tick at a time. At each step the rules see only what had already happened, and any order they place is filled by the tester’s own assumptions. MetaTrader includes a Strategy Tester for this purpose. The result is a list of trades and the figures drawn from it: net result, drawdown, number of trades, the share that won.

Every one of those figures depends on assumptions.

• The price history. Gaps or errors in the data produce trades that could never have happened.
• The detail. A test on bar opening prices is fast and coarse. A test on every tick is slower and closer to what an order would have met.
• The costs. A fixed spread and no slippage flatter any system that trades often or trades around news.
• The fills. A tester fills every order in full. A live market does not promise that.

Optimisation

Most rules have settings: the period of an average, the distance of a stop. Optimisation runs the backtest again and again with different settings and ranks the results. Suppose a system has two settings and each is tried at 50 values. That is 50 × 50 = 2,500 backtests, and the tester reports the best of them.

The best of 2,500 attempts is partly the best by luck. The more combinations are tried, the more likely it is that the winner fitted accidents of that stretch of history.

Overfitting

Overfitting is the name for rules that have been tuned to the noise in past prices instead of to anything that repeats. An overfitted system describes the past very well and says little about what comes next. Several signs point to it.

• Many settings, each adjusted until the result improved.
• A result that collapses when a setting is moved by one step.
• Few trades, or a result that rests on a handful of them.
• An equity curve that is almost a straight line.

Out-of-sample testing

The first defence is to keep some history back. The settings are chosen on one part, called in-sample, and then tested once on a part they have never seen, called out-of-sample.

A large fall in performance from the first part to the second is the mark of overfitting. The check works only once. If the settings are changed after looking at the out-of-sample result, that data has become in-sample too.

Walk-forward testing

Walk-forward testing repeats the same idea along the whole history. The settings are optimised on one window, tested on the stretch that follows, and then the window moves forward and the process starts again. The out-of-sample stretches are joined into one record.

That record shows how the method of choosing settings would have fared, not how one lucky set did. It is a harder test to pass, and it is still a test on the past.

Why a good backtest is not a forecast

A backtest answers one question: what would these rules have done on these prices under these assumptions? It cannot say that the future will resemble that history, that live costs will match the modelled ones, or that orders will fill as the tester filled them. A backtest shown by someone selling a system is also the one they chose to show.