Build and test a strategy Lesson 10 of 14
Strategies · Building and testing a strategy
Overfitting and backtest bias: 4,094 rules tested on SPY
How searching for the best rule creates results that vanish: 4,094 calendar rules and 23 moving-average lengths chosen on one half of SPY history, checked on the other.
On this page
Overfitting is what happens when a rule is shaped to fit the past so closely that it has learned the accidents of that particular history and not anything that will repeat. An overfitted strategy has an excellent backtest and no future. It is the most common reason a tested strategy disappoints, and it needs no dishonesty at all: it is enough to try many versions and keep the best.
This lesson shows the effect with rules that have no reason to work. We test 4,094 calendar rules on the first half of SPY's history, pick the winners, and follow them into the second half. Then we do the same with a rule that does have a reason behind it, the moving-average rule, and see what a sturdier result looks like.
What overfitting is
Every price history is a mix of two things: patterns that come from how markets and people behave, and noise that will never repeat in the same form. A rule with enough adjustable parts can fit both. The backtest cannot tell which one it fitted, because it only measures the fit.
The adjustable parts are everywhere. The length of an average. The level at which an indicator counts as "low". A filter that skips certain months. A stop distance. The start date of the test. Each one multiplies the number of possible strategies. Three settings with ten values each make 1,000 versions, and one of those 1,000 will be the best on any data, including random numbers.
Overfitting also happens without a computer search. A trader who looks at a chart, adjusts a rule until it "would have caught" the 2020 fall, and then adjusts it again for 2022, has run a search by hand. So has a research community: when thousands of people test ideas on the same price history and only the successes are published, the published record is the best of many tries. Harvey, Liu and Zhu counted more than 300 published factors and argued that most would not pass a test that accounted for the search.
An experiment with rules that should not work
To see overfitting cleanly, use rules nobody believes in. A calendar rule holds SPY in a fixed set of months each year and holds cash in the others: "own it in March, April, May, July, October, November and December; sit out the rest". There are 4,094 such rules, one for every choice of months except "none" and "all".
The test, stated in full:
- Data. SPY month-end closes, January 1995 to October 2026, 382 months. Monthly bars are the only ones the Finabase chart data carries for the full history.
- The split. The first half runs from January 1995 to November 2010. The second half runs from November 2010 to October 2026. Rules are ranked on the first half only.
- Trading. The account is in SPY for the chosen months and in cash otherwise, trading at the month-end close.
- Costs. 0.10% of the amount on every buy and sell.
- Left out. Dividends, interest on cash and taxes.
Buying and holding SPY returned 6.0% a year in the first half, with a deepest fall of 52.2%, and 12.3% a year in the second.
The best rules of the first half
| Months held | First half | Second half | Rank in the second half |
|---|---|---|---|
| Mar, Apr, May, Jul, Oct, Nov, Dec (the best) | 8.6% a year | 10.0% a year | 237 of 4,094 |
| Mar, Apr, May, Oct, Nov, Dec (second best) | 8.6% a year | 7.5% a year | 954 of 4,094 |
| Average of the 20 best | 7.9% a year | 9.0% a year | n/a |
| Buy and hold | 6.0% a year | 12.3% a year | n/a |
The winning rule beat buy and hold by 2.6 points a year for 16 years, with half the deepest fall: 25.8% against 52.2%. Presented alone, with a chart, it would look like a discovery. It has a story ready to go with it, too: avoid the summer and the weak months of January and February.
In the second half it earned 10.0% a year while simply holding earned 12.3%. The 20 best rules together went from 1.9 points ahead to 3.3 points behind.
Across all 4,094 rules:
- 205 beat buy and hold in the first half. That is 5% of them. The market fell by half twice in those years, so a rule could come out ahead just by being in cash at lucky times.
- 7 of those 205 also beat it in the second half. Passing the first test said almost nothing about passing the second.
- The average rule earned 2.4% a year, then 5.4%. The typical rule simply holds SPY half the time and gets about half the return.
The dots in the figure do slope upward, and it is worth knowing why. Rules that hold more months earn more in both halves, because the market rose in both. That is exposure, not skill. What the search added on top of exposure, the choice of which months, did not carry over.
The arithmetic of many tries
Suppose a rule with no edge has a 5% chance of looking good by luck. Test one such rule and there is a 5% chance of being fooled. Test 20 independent ones and the chance that at least one looks good is
With 100 tries it is above 99%. So the question to ask of any backtest is not "is this result good?" but "how many versions were tried before this one was shown?". Bailey, Borwein, López de Prado and Zhu make the same point: without the number of trials, a reported backtest result cannot be evaluated.
A rule with a reason: moving-average lengths
Now the same exercise with the rule from Moving average trend following: the 10-month rule, tested: hold SPY while its month-end close is above its average of the last N months, otherwise hold cash. There is a reason it might work, since long declines spend most of their time below the average. There is one setting, N. We try every N from 2 to 24 on the same two halves.
| Length | First half | Second half |
|---|---|---|
| 2 months | 3.3% a year | 5.5% a year |
| 6 months | 6.3% a year | 8.4% a year |
| 10 months | 9.4% a year | 7.5% a year |
| 12 months (best, first) | 9.7% a year | 8.4% a year |
| 13 months (best, second) | 9.3% a year | 9.5% a year |
| 18 months | 9.4% a year | 9.3% a year |
| 24 months | 9.4% a year | 7.2% a year |
| Buy and hold | 6.0% a year | 12.3% a year |
Three things separate this from the calendar rules.
- The first-half result is a plateau, not a spike. Every length from 8 to 24 months returned between 8.5% and 9.7% a year. The "best" length, 12, is not special; its neighbours did nearly as well. A calendar rule changes character completely when one month is added or dropped.
- The ordering partly survived. Very short averages were poor in both halves. That is a pattern with a cause: short averages trade more and get whipsawed more.
- The picked setting still faded. The 12-month rule went from 9.7% a year to 8.4%, and in the second half every one of the 23 lengths trailed buy and hold. The rule's edge over holding in 1995–2010 came from two bear markets that the later period did not repeat.
The third point is the humbling one. Even a simple rule with a reason, one setting and a plateau gave a rosier picture in the half where it was chosen. Overfitting is a matter of degree, and no test is free of it.
Other ways a backtest flatters
Overfitting is one of a family of biases. Backtesting basics: biases, costs and an honest SPY test covers them in more detail; in short:
| Bias | What goes wrong | A sign of it |
|---|---|---|
| Look-ahead | The rule uses information that was not known at the time | A rule that acts on a day's high or low, or on restated figures |
| Survivorship | The test uses only stocks that still exist or are in an index now | A stock list built from today's members |
| Selected period | The start and end dates were chosen after seeing the results | A test that begins just after a crash |
| Ignored costs | No spread, commission or slippage | A rule with many trades and a small average gain |
| Too few trades | The result rests on a handful of events | Fewer than about 30 trades |
| Too many settings | The rule was tuned until it fitted | Odd, exact values such as "17 days" or "RSI below 23.5" |
How to keep a test honest
None of these removes the problem. Each makes it smaller or easier to see.
- Start from a reason. Decide why a rule should work before testing it. A reason limits the search and gives something to check when the rule stops working.
- Keep the settings few. Each one is another way to fit noise.
- Show the neighbours. Report the settings on either side of the chosen one. A plateau is reassuring. A lone peak is a warning.
- Hold data back. Choose on one period and judge on another that the choice never saw. Out-of-sample and walk-forward testing, worked on SPY shows how, and why it can only be done once.
- Count the tries. Keep a list of every version tested. The more there are, the more a good result should be discounted.
- Expect less. Treat a backtest as an upper bound. Costs, mistakes and the search itself all push the live result below it.
How Finabase does it
The strategy lessons in this section follow the rules above: each states its data, its costs and what is left out, splits its test period at least once, and shows neighbouring settings beside the one it describes.
Finabase's own tools make searching easy, which is exactly why this lesson matters when you use them.
- Backtest. The Backtest tool trades every signal of the indicators you switch on, on one symbol, over the range you choose, and lets you edit each indicator's parameters. Each signal trades a fixed $10,000 and no trading cost is charged. Changing a parameter and re-running until the gain looks good is a search of the kind described here. The tool does not count your tries; you have to.
- Change the range. The simplest check available in the tool is the range setting. A setting that looks good over one year should be looked at over a different range before it is believed.
- Strategy. The Strategy tool replays if/then rules over two years only. A rule tuned to two years of one market has seen very little, so treat a good result there as a description of those two years.
No Finabase tool tells you which settings to use or what to buy.
Frequently asked questions
What is overfitting in trading?
Overfitting is tuning a strategy until it matches past prices so closely that it has fitted their random noise. The backtest looks excellent, and the result does not carry forward, because the noise does not repeat.
How can I tell if a backtest is overfitted?
Look for many settings, oddly exact values, a result that collapses when a setting is nudged, few trades, and no test on data that was held back. Ask how many versions were tried. If the answer is unknown, assume the result is the best of many.
Is optimizing a strategy always wrong?
No. Choosing a sensible setting is part of building any rule. The damage comes from treating the best of many tries as if it were the only try, and from judging the rule on the same data that chose it.
Does a longer backtest prevent overfitting?
It helps, because more data holds more kinds of market and leaves less room for luck. It does not prevent it. A wide enough search will find something that fits any history, however long.
Summary
Overfitting comes from searching: try enough rules on the same history and the best will look like a discovery whether or not anything real is behind it. Of 4,094 calendar rules on SPY, 205 beat buy and hold from 1995 to 2010, and 7 of those kept doing so through 2026. The top 20 swung from 1.9 points a year ahead to 3.3 behind. A moving-average rule, with a reason and one setting, held up better: its results formed a plateau across lengths, yet its chosen setting still faded. Good practice is a reason first, few settings, neighbours shown, data held back and an honest count of how many versions were tried.
Check your understanding
5 questions on this lesson. Each answer comes with the reason, and nothing is scored.
A researcher tests 500 versions of a rule and reports the best one, which beat the market by 3 points a year. What is the right reading?
- The rule has a proven edge of 3 points a year
- The result is the best of 500 tries and should be heavily discounted until it is checked on data the search never saw
- The rule is certainly worthless
Show the answer
B. The result is the best of 500 tries and should be heavily discounted until it is checked on data the search never saw The best of many tries looks good whether or not any version has an edge. The number of versions tried is part of the result. Only data held back from the search can show how much is left.
Of 4,094 calendar rules on SPY, 205 beat buy and hold in 1995–2010 and only 7 of those also did in 2010–2026. What does this show?
- Passing the first period said almost nothing about the second
- Seven calendar rules have a real edge
- Calendar effects became stronger over time
Show the answer
A. Passing the first period said almost nothing about the second The rules were chosen by their fit to the first period, and that fit was mostly accident. Even the seven survivors are what luck would leave behind from 205 tries.
Moving-average lengths from 8 to 24 months all returned between 8.5% and 9.7% a year in the first half. Why is that more reassuring than one length standing far above the rest?
- It means the best length can be used with confidence
- A plateau shows the result does not depend on one exact setting, which is what fitted noise usually looks like
- It proves the rule will beat buy and hold
Show the answer
B. A plateau shows the result does not depend on one exact setting, which is what fitted noise usually looks like Noise produces lone peaks: nudge the setting and the result collapses. A broad plateau suggests the result comes from the idea, not the number. It still faded in the second half, where every length trailed buy and hold.
Which habit does the most to limit overfitting?
- Adding filters until the losing trades disappear
- Testing on as short and recent a period as possible
- Deciding why a rule should work before testing it, keeping the settings few, and holding data back
Show the answer
C. Deciding why a rule should work before testing it, keeping the settings few, and holding data back A reason limits the search, few settings leave less room to fit noise, and held-back data shows what survives. Adding filters to remove past losses is overfitting itself.
With 20 independent rules that have no edge, each with a 5% chance of looking good by luck, how likely is it that at least one looks good?
- 5%
- About 64%
- 100%
Show the answer
B. About 64% The chance that none passes is 0.95 multiplied by itself 20 times, about 36%, so the chance that at least one does is about 64%. More tries push it toward certainty.
Exercise
One task to do in Finabase. Nothing is graded or saved.
Open the sample backtest at its 1 year range and write down the Gain / max cost and the Win rate. Now switch Range to 6 months, then 2 years, then 5 years, writing down the same two numbers each time. Do the four results sit close together or jump around? Then count: how many versions of this backtest have you now looked at, and which one would you be tempted to show someone?
Open the sample backtestWhat you should find
You have run four versions of one test. If you would show only the best, you have done a small search, and with enough tries one version always looks good. Results that stay similar across ranges are more reassuring; results that swing widely mostly describe the stretch of history each range happens to cover. The tool does not count your tries, so the count is yours to keep.
Sources and further reading
- David H. Bailey, Jonathan M. Borwein, Marcos López de Prado and Qiji Jim Zhu, “Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance,” Notices of the American Mathematical Society 61(5), 2014
- Campbell R. Harvey, Yan Liu and Heqing Zhu, “…and the Cross-Section of Expected Returns,” Review of Financial Studies 29(1), 2016 — how many published findings survive a multiple-testing correction
- Halbert White, “A Reality Check for Data Snooping,” Econometrica 68(5), 2000
- Finabase chart data — SPY monthly bars, January 1995 to October 2026; rules and results computed by Finabase
