Build and test a strategy Lesson 5 of 14
Strategies · Building and testing a strategy
Choosing a benchmark: what to compare a strategy against
How to pick a fair yardstick for a strategy: one SPY rule judged against cash, buy and hold and a matched mix, and a QQQ rule that beats the wrong benchmark.
On this page
A return means nothing alone. "The strategy made 8% a year" is good news if the alternative made 4% and bad news if it made 12%. A benchmark is the alternative: what the same money would have earned in a simple, available option over the same period. Choosing it is part of describing a strategy, and choosing it badly is the easiest way to make a mediocre strategy look clever.
This lesson judges one rule on SPY against four different yardsticks and gets four different verdicts. It then shows a rule on QQQ that beats "the market" by more than a point a year without adding anything. Along the way it sets out what makes a benchmark fair.
What a benchmark is for
A strategy is a decision to do something other than the simplest thing. The benchmark is the simplest thing. Comparing the two answers the only question that matters about the extra effort: was it worth it?
That makes the benchmark depend on the question. There are three common ones.
| Question | Benchmark |
|---|---|
| Did it make money at all? | Cash |
| Did the rule beat owning the same thing? | Buying and holding what the strategy trades |
| Did the timing add anything? | A constant mix with the strategy's average exposure |
Professional performance measurement adds a list of properties a valid benchmark should have. The list associated with Jeffery Bailey and his colleagues Thomas Richards and David Tierney is the usual reference: a benchmark should be unambiguous, investable, measurable, appropriate to the strategy, and specified in advance. In plain terms:
- Same assets. A strategy that trades technology stocks is compared with technology stocks, not with the whole market.
- Similar risk. A strategy that is in cash a quarter of the time takes less risk than one always invested. A comparison that ignores this judges two different things.
- Same dates. Both sides start and end on the same day.
- Same treatment of costs and income. If the strategy pays trading costs, an investable benchmark is used, not a cost-free index. If dividends are left out of one, they are left out of both.
- Really available. An index fund you could have bought, not an index number or an average of other people's results.
- Chosen first. Picking the benchmark after seeing the results is a form of overfitting.
One strategy, four verdicts
The strategy is the 10-month rule from Moving average trend following: the 10-month rule, tested: hold SPY while its month-end close is above its 10-month average, otherwise hold cash.
The test, stated in full:
- Data. SPY month-end closes, November 1993 to October 2026, 396 months.
- Trading. At the month-end close. The rule is fully in SPY or fully in cash. The constant mixes are rebalanced to their weights every month.
- Costs. 0.10% of the amount traded, for the rule and for the mixes' rebalancing alike.
- Left out. Dividends, interest on cash and taxes, on every line. Leaving out interest makes cash look worse than it was; leaving out dividends makes SPY look worse than it was.
| Yardstick | Return per year | Deepest fall | Verdict on the rule (7.8% a year, −23.8%) |
|---|---|---|---|
| Cash | 0.0% | 0.0% | Made money |
| SPY held throughout | 8.9% | −52.2% | 1.1 points behind, with half the fall |
| 76% SPY, 24% cash, rebalanced monthly | 6.9% | −42.5% | 0.9 points ahead, with a much smaller fall |
| 60% SPY, 40% cash, rebalanced monthly | 5.5% | −35.1% | 2.3 points ahead, with a smaller fall |
Each row is true, and each answers a different question.
Against cash the rule looks excellent, and the comparison is almost useless. Nearly anything that held US stocks over these 33 years beat cash. A strategy described only as "up 1,100%" is being compared with cash without saying so.
Against SPY held throughout the rule trails by 1.1 points a year. This is the comparison most people mean by "did it beat the market". It is fair on assets and dates. It is not fair on risk: the rule was invested 76% of the time and its deepest fall was 23.8% against 52.2%.
Against the 76/24 mix the rule is ahead. This is the benchmark that isolates timing. The rule was in SPY 76% of the time, so its natural competitor is a portfolio that is 76% in SPY all of the time. Both have the same average exposure. The only difference is when the rule was out. Over the whole period, being out at the times the rule chose was worth 0.9 points a year and a far smaller deepest fall than the mix's 42.5%.
Against 60/40 it is further ahead. This asks a different question again: how does it compare with a standard cautious allocation of the same asset?
Return for the risk taken
One way to bring return and risk into a single figure is to divide the return per year by the deepest fall.
| Line | Return per year ÷ deepest fall |
|---|---|
| 10-month rule | 0.33 |
| SPY held throughout | 0.17 |
| 76% SPY, 24% cash | 0.16 |
On this measure the rule earned about twice as much per unit of its worst loss. Measures like this one, or the Sharpe ratio, are what "risk-adjusted" means. They depend heavily on the period, as the next section shows.
The verdict changes with the period
| Period | 10-month rule | SPY held | 76/24 mix |
|---|---|---|---|
| November 1993 – February 2009 | 8.0% a year | 3.1% a year | 2.6% a year |
| February 2009 – October 2026 | 7.7% a year | 14.1% a year | 10.7% a year |
Through the low of February 2009 the rule beat both benchmarks by about five points a year. Since then it has trailed the matched mix by 3.0 points a year. The timing that added value in two long bear markets has subtracted value in every year without one. A benchmark comparison over one long period can hide two opposite results. Splitting the period is part of comparing honestly.
Beating the wrong benchmark
Now apply the same 10-month rule to QQQ, the fund that tracks the Nasdaq-100, and compare it with "the market" in the usual sense, SPY. QQQ month-end closes, January 2000 to October 2026, the same costs, dividends left out.
| Line | Return per year | Deepest fall |
|---|---|---|
| 10-month rule on QQQ | 7.8% | −36.9% |
| SPY held throughout | 6.5% | −52.2% |
| QQQ held throughout | 7.9% | −81.1% |
| 71% QQQ, 29% cash, rebalanced monthly | 6.2% | −67.5% |
Against SPY, the rule "beat the market by 1.3 points a year with a smaller deepest fall". That sentence is accurate and misleading. The rule traded a different fund from the benchmark. Over these years QQQ itself returned more than SPY, so anything that held QQQ most of the time inherited that gap.
Against QQQ held throughout, the rule's return is 0.1 points lower. On return alone it added nothing. What it did change is risk, and that part is real: a deepest fall of 36.9% against 81.1%, and a clear lead over the matched 71/29 mix.
From February 2009 the picture is plainer still: the rule on QQQ returned 12.5% a year, SPY 14.1% and QQQ 20.0%. The rule trailed even the "wrong" benchmark.
This is the most common benchmark error, and it appears in many forms:
- a stock-picking record compared with an index of different stocks;
- a strategy using borrowed money compared with an unleveraged fund;
- a small-company portfolio compared with a large-company index;
- a portfolio that includes dividends compared with a price-only index.
In each case part of the gap comes from what was held, not from any decision the strategy made. Sharpe's 1992 paper made this the basis of style analysis: first find the mix of index funds that behaves like the strategy, then ask whether the strategy beat that mix.
Choosing a benchmark in practice
- Write down what the strategy can hold. That defines the asset side: the same fund, the same index, or the same list of stocks, equally weighted.
- Note its average exposure. If it is in cash part of the time, add a constant mix with that exposure as a second benchmark.
- Use something investable. A fund with a price history, tested with the same cost assumption.
- Match the dates and the income. Same first and last bar, dividends in both or neither.
- Fix it before testing, and report every benchmark you looked at, not the one that flatters.
- Compare the falls as well as the returns, and split the period at least once.
For a strategy that selects among many stocks, the fair benchmark is usually the whole list, held in equal weights. For one that rotates among several funds, it is an equal-weight portfolio of those funds. A broad index fund remains a useful second line in every case, because it is what most people would otherwise own, as Index investing: passive vs. active funds and what the data shows explains.
How Finabase does it
- Backtest. The Backtest tool reports the gain of the signals you choose on one symbol, in dollars and as a percentage of the most capital used, with a fixed $10,000 per signal and no trading cost. It does not print a benchmark beside the result, so the comparison is yours to make. The natural one is the symbol's own change over the same range, which the symbol page and chart show.
- Strategy. The Strategy tool replays if/then rules over two years from $10,000 and shows the return and account value. Compare that return with what the funds named in the rule did over the same two years, not only with a broad index.
- Chart comparison. The chart can draw other symbols against the main one on a percentage scale. The link above opens SPY with QQQ, which shows at a glance why a benchmark from a different part of the market misleads.
- This section's tests compare every strategy with buying and holding what it trades, over the same bars and with dividends left out of both, and say when a second yardstick is needed.
No Finabase tool tells you what to buy or sell.
Frequently asked questions
What is a good benchmark for a trading strategy?
Buying and holding the same thing the strategy trades, over the same dates and with the same cost assumptions. If the strategy is often in cash, add a constant mix with its average exposure so that the timing can be judged separately.
Is the S&P 500 always the right benchmark?
No. It is the right benchmark for a strategy that holds large US stocks. For a strategy in technology stocks, small companies, bonds or several asset classes, an S&P 500 comparison mostly measures the difference between those assets and the index.
Should a benchmark include dividends?
It should match the strategy. If the strategy's results include dividends, the benchmark must too; if they are left out of one, leave them out of both and say so. Comparing a result with dividends against a price-only index overstates the result.
What if a strategy has a lower return than its benchmark but smaller falls?
Then it is a different trade-off, not simply a failure. Compare return per unit of risk, look at a benchmark with similar risk such as a constant mix, and decide whether the smaller falls are worth the return given up. The answer depends on the investor, not the backtest.
Summary
A benchmark is the simple alternative a strategy has to justify itself against, and the choice of benchmark decides the verdict. The 10-month rule on SPY made money against cash, trailed holding SPY by 1.1 points a year with half the deepest fall, and beat a constant mix with its own average exposure by 0.9 points, with all of that advantage earned before 2009. The same rule on QQQ beat SPY and added nothing to QQQ's own return. A fair benchmark holds the same assets, takes similar risk, covers the same dates and costs, can really be bought, and is chosen before the results are known.
Check your understanding
4 questions on this lesson. Each answer comes with the reason, and nothing is scored.
A rule that trades only a technology fund returned 12% a year. The S&P 500 returned 9% and the technology fund itself 13%. What did the rule add?
- Three points a year over the market
- Nothing on return: it trailed the fund it trades by a point, and the gap over the S&P 500 came from holding technology
- Four points a year
Show the answer
B. Nothing on return: it trailed the fund it trades by a point, and the gap over the S&P 500 came from holding technology The fair benchmark holds the same assets. The lead over the S&P 500 belongs to the fund, which anyone could have bought and held. Against that fund the rule gave up a point a year.
A strategy is invested 70% of the time and in cash otherwise. Which benchmark isolates whether its timing helped?
- Cash
- A portfolio that is always 70% invested in the same fund
- The best-performing fund of the period
Show the answer
B. A portfolio that is always 70% invested in the same fund A constant 70% mix has the same average exposure, so the only difference left is when the strategy chose to be out. Cash is beaten by almost anything, and a fund picked afterwards is hindsight.
The 10-month rule on SPY returned 7.8% a year with a deepest fall of 23.8%. Holding SPY returned 8.9% with a deepest fall of 52.2%. Which statement is fair?
- The rule failed, because its return was lower
- The rule gave up 1.1 points a year for less than half the deepest fall; whether that is worth it depends on the investor
- The rule is better, because its fall was smaller
Show the answer
B. The rule gave up 1.1 points a year for less than half the deepest fall; whether that is worth it depends on the investor The two lines differ in risk as well as return, so neither number settles it alone. A benchmark comparison should state both and leave the trade-off to the person who has to live through the falls.
Why should the benchmark be chosen before the test is run?
- Because choosing it afterwards lets you pick whichever comparison flatters the result
- Because benchmarks change every year
- Because the test runs faster
Show the answer
A. Because choosing it afterwards lets you pick whichever comparison flatters the result With several possible yardsticks, one of them will usually make the strategy look good. Selecting it after seeing the results is a search, the same mistake as tuning a rule to past data.
Exercise
One task to do in Finabase. Nothing is graded or saved.
Open the sample strategy. Read its rule lines (if, then, else), then note the Return, the number of trading days, and the date and price of the buy in the Trades table. The page prints no benchmark. What is the fair one for this rule? Work out its return over the same days from that buy price and the latest SPY price. Did the rule add anything?
Open the sample strategyWhat you should find
Both branches of the sample hold SPY, so the fair benchmark is SPY bought on the same day and held. Its return matches the rule's: the rule added nothing to holding the fund. Setting the Return beside SPY's one-year or two-year figure from a quote page would break the same-dates rule and show a gap that is only a difference in start date.
Sources and further reading
- Jeffery V. Bailey, “Are Manager Universes Acceptable Performance Benchmarks?,” Journal of Portfolio Management 18(3), 1992, pp. 9–13 — the properties of a valid benchmark
- William F. Sharpe, “Asset Allocation: Management Style and Performance Measurement,” Journal of Portfolio Management 18(2), 1992, pp. 7–19 — judging a manager against a matching mix of indexes
- S&P Dow Jones Indices — SPIVA scorecards, funds compared with benchmarks of the same style
- Finabase chart data — SPY monthly bars, 1993 to October 2026, and QQQ monthly bars, 1999 to October 2026; rules and results computed by Finabase
