Guide · 6 min read · Updated 17 November 2026
Overfitting and out-of-sample testing, in plain English
Overfitting is when a strategy has learned the noise in past prices instead of a pattern that repeats. It looks brilliant in the backtest and disappoints with real money. It is the main reason backtests and live results disagree.
How overfitting happens
Every time you tweak a setting to make the past look better, such as trying a 17-day average instead of 20, or adding one more filter, you fit the rules a little more tightly to that particular history. Try enough combinations and something will look excellent by chance alone.
Signs a strategy may be overfitted:
- Many rules or settings relative to the number of trades.
- Oddly specific numbers (RSI 27, a 13-day average) with no reason behind them.
- Performance collapses when a setting moves slightly.
- Results depend on a few very large trades.
Out-of-sample testing
The fix is to keep part of the data hidden. Split the history by time into a training period and a test period. Build or describe the strategy using only the training period, then run it once on the test period.
If the test period looks broadly like the training period, the idea has passed one honest check. If it falls apart, the training result was probably luck or fitting.
How we use it
Every AlphaEdgeCheck result is split by time: the first 70% is training and the last 30% is test, and both are shown side by side with the full period. The engine does not optimise any settings, so the split shows whether the idea held up on data it was not described against. Details are on the methodology page.
Limits worth knowing
One train/test split is a single experiment. If you look at the test result, change the rules and test again, the test period quietly becomes training data. And a strategy that passed in one market regime can still fail in the next. Forward tracking on new data is the strongest check of all.
Common questions
- What is a good train/test split?
- A common choice is 70/30 or 80/20 by time, never shuffled, because markets change over time and a random split leaks future information into training.
- Is walk-forward testing better than one split?
- Walk-forward analysis repeats the train/test step across several windows, which gives more than one out-of-sample result. It is more thorough, but the principle is the same: judge the rules on data they were not fitted to.
Research, not advice. This guide is general education. It does not recommend buying or selling anything. Read the risk disclosure.