Out-of-Sample Testing
Out-of-sample testing checks a strategy on data that was not used to develop or optimise it, to see whether its edge is real or the product of fitting the past.
Explained
A common approach reserves the last 20–30% of data. Walk-forward analysis repeats this across rolling windows. A big drop out of sample signals overfitting.
Example
A strategy earns 25% a year in 2010–2019 (in sample) and 4% in 2020–2024 (out of sample).
The gap suggests much of the backtest was overfit.
Common mistakes
Peeking at out-of-sample data during development.
Re-optimising after a poor out-of-sample result.
Using too short an out-of-sample period.
Questions traders ask
How much data to hold out?
Often 20–30%, covering different conditions.
What is walk-forward testing?
Rolling optimisation and out-of-sample testing across many periods.