Skip to content
Menu
Finabase Learn

Build and test a strategy Lesson 14 of 14

Strategies · Building and testing a strategy

Why trading strategies stop working

The five reasons a tested strategy fades, a rule on SPY and the small-company premium measured block by block, and how to tell a bad stretch from a broken idea.

On this page
  1. The five reasons
  2. Case one: a trend rule, block by block
  3. Case two: the small-company premium
  4. Telling a bad stretch from a broken idea
  5. How Finabase does it
  6. Frequently asked questions
  7. Summary
  8. Check your understanding
  9. Exercise

A strategy that tested well and then disappoints is the normal case, not the exception. The backtest shows years of steady advantage; real money goes in; the advantage shrinks, vanishes or reverses. The person following it is left with the hardest question in trading: is this an ordinary bad stretch, or is the idea finished?

There are only a handful of reasons a strategy stops working, and they call for different responses. This lesson sets them out, then looks at two real cases block by block: a simple trend rule on SPY whose whole advantage came before 2009, and the small-company premium, one of the best known findings in finance, measured on live funds since 2000. It closes with what can and cannot be told from inside a losing stretch.

The five reasons

1. It never worked

The most common reason is that the edge was never there. Test enough rules on the same history and some will look excellent by chance. Change the settings until the result improves and the rule ends up describing the past, not the market. This is overfitting, covered in Backtesting basics: biases, costs and an honest SPY test.

Harvey, Liu and Zhu counted 316 factors that academic papers had claimed could explain stock returns, and argued that with so many tries the usual standard of evidence is far too weak: many of the published findings are likely false. If trained researchers with peer review produce false positives at that rate, a private backtest tuned over a weekend has little chance.

A strategy of this kind does not stop working. It simply meets data it was not fitted to.

2. The edge was real and was competed away

Some patterns exist because of a genuine mistake in prices. Once enough money knows about the mistake and trades on it, the trading itself removes it. Buyers push the cheap thing up before the pattern can pay.

McLean and Pontiff measured this directly. They took 97 patterns in stock returns that academic studies had documented and checked each one in two later periods. After the original sample ended but before the paper was published, returns were 26% lower. After publication they were 58% lower. The first drop is mostly reason 1: results flatter their own sample. The further drop after publication is investors learning about the pattern and trading it away.

Competition also creates a new danger. When many funds hold the same positions for the same reason, one fund's forced selling pushes prices against all the others. In the second week of August 2007 a group of quantitative stock funds running similar strategies suffered large losses within days and then largely recovered, with little movement in the overall market. Khandani and Lo traced it to exactly this kind of crowding.

3. The market's structure changed

Some edges depend on how trading works: its costs, its rules, who takes part. When those change, the edge can go permanently.

  • US stock prices moved from fractions to decimals by April 2001, on the SEC's order. The minimum price step fell from one-sixteenth of a dollar to one cent, spreads narrowed, and strategies that lived on wide spreads lost their margin.
  • Commissions at the large US retail brokers fell to zero in October 2019. A short-term rule that was unprofitable after a $5 commission could become viable; one whose edge came from others' costs could lose it.
  • Calendar patterns that depended on tax-driven selling or on month-end cash flows weakened as they became known and as markets changed (Calendar effects in stocks: January, sell in May and day of week).

A backtest that spans such a change is really two tests joined together, and only the later part describes the market that exists now.

4. It needs conditions that have not come back

Many strategies are not always-on edges. They pay in one kind of market and cost money in another. Trend following pays in long declines and loses in sharp reversals. Buying dips pays in rising markets and is ruinous in a real bear market. If the favorable condition has been absent for ten years, the strategy looks dead, and nothing about it has changed.

This is the hardest case, because from the inside it is indistinguishable from reasons 1 to 3.

5. Ordinary bad luck

Even a real, stable edge produces long losing stretches by chance. A rule that makes 25 trades in 33 years has very few throws of the dice. A run of four or five bad ones is expected at some point, and when it comes it can last years.

Case one: a trend rule, block by block

The rule is the 10-month moving average rule on SPY from Moving average trend following: the 10-month rule, tested: hold the fund while its month-end close is above its 10-month average, otherwise hold cash.

  • Data. SPY month-end closes; the test runs from November 1993 to October 2026.
  • Trading and costs. Trades at the signal's month-end close, 0.10% per buy and per sell.
  • Left out. Dividends, interest on cash and taxes.

Over the whole period the rule returned 7.8% a year against 8.9% for buying and holding, with a deepest fall of 23.8% against 52.2%. Cut into five-year blocks, the return gap looks like this.

10-month rule on SPY minus buy and hold, return per year, by five-year block −2.6 1996–00 +4.8 2001–05 +7.7 2006–10 −2.8 2011–15 −5.2 2016–20 −5.2 2021–25 Percentage points a year. Above zero: the rule beat holding. 0.10% per trade, no dividends.
The 10-month rule on SPY minus buy and hold, in percentage points a year, by five-year block. 0.10% per trade, dividends not included.
Block 10-month rule Buy and hold Gap (points a year)
1996–2000 13.8% 16.4% −2.6
2001–2005 3.7% −1.0% +4.8
2006–2010 7.9% 0.2% +7.7
2011–2015 7.4% 10.1% −2.8
2016–2020 7.7% 12.9% −5.2
2021–2025 7.6% 12.8% −5.2

Two blocks out of six carry the whole case for the rule, and they are the two that contain the bear markets of 2000 to 2002 and 2007 to 2009. In the other four it trailed. Measured as a running total, the rule's value relative to buy and hold peaked in February 2009 at 2.01, twice as much money as holding, and has fallen since to 0.73.

Which of the five reasons is this? The paper that made the rule famous was published in 2007, so competition is a tempting answer. It does not fit well. A rule that moves a diversified fund to cash once a year is not exploiting a mispricing that other traders can close. The better reading is reason 4. The rule is paid by long declines, and since 2009 the US market has had sharp falls, in 2018, 2020, 2022 and 2025, and only one, in 2022, that lasted most of a year; it fell half as far as the declines of 2000 and 2008. Its own return barely changed: 7.4% to 7.9% a year in each of the last four blocks. What changed was the thing it is compared with.

That reading could be wrong. Markets may now recover faster than they used to, because of how central banks respond or because dip-buying has become a habit, and that would be reason 3. The data cannot separate the two. That is the honest position, and it is uncomfortable.

Case two: the small-company premium

In 1981 Rolf Banz reported that the shares of small US companies had earned more than those of large ones over 1936 to 1975, by more than their extra risk seemed to explain. The "size effect" became one of the most cited findings in finance, and funds were built to collect it.

IWM, a fund that tracks the Russell 2000 index of small companies, has traded since 2000. SPY holds the large ones. The comparison uses month-end prices from June 2000 to October 2026 and leaves out dividends, which have generally been a little higher on SPY, so the table slightly flatters the small-company fund.

Small companies (IWM) minus large (SPY), return per year, by five-year block +7.9 2001–05 +3.0 2006–10 −2.6 2011–15 −1.2 2016–20 −8.1 2021–25 Percentage points a year, month-end prices, dividends not included.
Small companies (IWM) minus large (SPY), in percentage points a year, by five-year block. Month-end prices, dividends not included.
Block IWM (small) SPY (large) Gap (points a year)
2001–2005 6.9% −1.0% +7.9
2006–2010 3.2% 0.2% +3.0
2011–2015 7.6% 10.1% −2.6
2016–2020 11.7% 12.9% −1.2
2021–2025 4.7% 12.8% −8.1

Over the full 26 years the two funds ended almost level: 6.6% a year for IWM and 6.5% for SPY. An investor who bought the small-company fund in 2000 for its premium received all of it in the first ten years and gave all of it back in the next fifteen.

Here the candidate reasons are different. The original finding has been questioned on the grounds of reason 1: much of it came from the very smallest stocks, which are costly to trade, and from a few decades. Reason 2 applies as well, since the effect was published in 1981 and weakened in the US data soon after. And reason 4 has a clear form: the last block is the period in which a handful of very large technology companies led the market. A reader does not have to choose. Several reasons usually act together, and a strategy with three plausible causes of decay deserves less confidence than one with a single cause that might reverse.

Telling a bad stretch from a broken idea

There is no test that settles it. There are checks that separate a strategy behaving as its own history says it can from one doing something new.

Compare the losing stretch with the backtest's own. The SPY rule trailed buy and hold in 200 of the 336 five-year windows in its test, 60% of them. Its worst five-year window lagged by 8.9 points a year, ending in February 2014, and no five-year window since the one ending September 2013 has been ahead. So "it has trailed for five years" was always the likely state, and is no evidence of anything. A shortfall well outside the worst the backtest ever showed would be.

Check the mechanism, not the result. A strategy with a stated reason for working can be checked against that reason. The trend rule exists to limit long declines. In 2022, the one extended fall since 2009, its deepest loss was 23.8% against 24.8% for holding: it did almost nothing. That is a more worrying fact than the 17 years of lagging returns, because it concerns the one job the rule has.

Look for a structural change with a date. Decimal pricing, zero commissions, a rule change, the launch of funds that trade the same idea. If the decay begins at such a date, the case for reason 3 or 2 is stronger, and those do not reverse.

Ask how many independent bets the record contains. Twenty-five trades in 33 years cannot distinguish a modest edge from none. A short-term rule with hundreds of trades a year reveals decay far sooner.

Write the stopping rule first. The time to decide what would count as broken is before the money goes in, when the backtest's worst fall and longest lag are just numbers on a page. "Stop if the fall exceeds one and a half times the worst in the test" is crude, and far better than deciding in the middle of the fall.

Sign Points toward
Never tested on data outside the fitting period It never worked
Edge shrank steadily after it became widely known Competed away
Edge ended at a dated change in costs or rules Structural change
Own return steady; the comparison ran ahead Conditions absent
Shortfall within the range of its own backtest Bad stretch, so far
Failed at the specific job it was built to do Something is broken

How Finabase does it

Finabase's tools can show whether a result holds across periods. They do not judge a strategy or say when to stop one.

  • Backtest lets you change the Range of a test from one day to the full history and re-runs it. Running the same signals over a recent range and an older one is a quick version of the block tables above. Remember how it differs from the tests on this page: each signal buys or shorts a fixed $10,000 and there are no trading costs, so a rule that trades often looks better there than it would in an account.
  • Strategy replays if/then rules over the last two years only. That is the wrong tool for the question "did this ever work", because two years is one kind of market. It is a reasonable tool for "how is this behaving now".
  • The Comparison tab of a symbol page sets one symbol beside others. The link above puts IWM beside SPY, the pair in the second table.

Both tools are described in Backtest indicator signals and rule-based strategies on Finabase.

Frequently asked questions

Why do strategies work in backtests and fail in live trading?

Most often because the backtest result was partly fitted to the data it was tested on, so it overstated the edge from the start. Costs, slower fills and the trader's own deviations take more. Published research finds stock-return patterns earn about a quarter less on data outside their original sample even before anyone trades on them.

How long should a strategy be given before it is judged?

Long enough to cover the kind of market it depends on, which for slow strategies can be more than a decade. The 10-month rule on SPY trailed buy and hold in 60% of all five-year windows of its own backtest, so five poor years say almost nothing about it. A rule that trades daily can be judged in months.

Does a strategy stop working once it is published?

It usually weakens and rarely vanishes overnight. McLean and Pontiff found returns to 97 published patterns were 58% lower after publication. Edges that are compensation for bearing a risk tend to survive better than those that came from a pricing mistake.

Can a strategy start working again?

Yes, when the reason was absent conditions and not a vanished edge. Trend rules lag for years between long declines and then earn their keep in one. A strategy whose edge was competed away or removed by a rule change does not come back.

Summary

Strategies fade for five reasons: the edge was an artifact of the test, it was real and was competed away, the market's structure changed, the conditions it needs have not recurred, or chance. On SPY the 10-month rule earned its entire advantage over buy and hold in two five-year blocks ending in 2010 and trailed in four others, most plausibly because no long decline has occurred since. The small-company premium paid 7.9 points a year in 2001 to 2005 and cost 8.1 in 2021 to 2025, ending level over 26 years. From inside a losing stretch these causes look alike. What can be done is to compare the shortfall with the strategy's own tested history, check whether it still does the specific job it was built for, look for a dated change in the market, and decide in advance what result would mean it is broken.

Check your understanding

4 questions on this lesson. Each answer comes with the reason, and nothing is scored.

  1. A study of 97 published stock-return patterns found returns 26% lower on later data before publication, and 58% lower after publication. What explains the first drop?

    1. Investors trading the pattern away
    2. Results partly fitted to their own sample, which overstated the edge
    3. Higher trading costs in later years
    Show the answer

    B. Results partly fitted to their own sample, which overstated the edge Before publication few investors knew of the pattern, so the fall on new data mostly reflects that the original result flattered its own sample. The further drop after publication is the part that fits investors learning of the pattern and competing it away.

  2. The 10-month rule on SPY returned between 7.4% and 7.9% a year in each five-year block from 2006 to 2025, yet it went from 7.7 points ahead of buy and hold to 5.2 points behind. What changed?

    1. The rule’s own return collapsed
    2. Trading costs rose sharply
    3. Buy and hold’s return, because no long decline occurred after 2009
    Show the answer

    C. Buy and hold’s return, because no long decline occurred after 2009 The rule’s return was steady. Buy and hold earned 0.2% a year in 2006–2010 and about 13% a year in 2016–2025. A rule that is paid by long bear markets looks finished when there are none, even though nothing about it has changed.

  3. A strategy has trailed its benchmark for five years. In its own backtest it trailed in 60% of all five-year windows. What does the five-year lag show?

    1. Very little; it is the strategy’s usual state in its own history
    2. That the edge has been competed away
    3. That the backtest was overfitted
    Show the answer

    A. Very little; it is the strategy’s usual state in its own history A losing stretch is evidence only when it is unlike the losing stretches the test already contained. A shortfall well beyond the worst in the backtest, or a failure at the specific job the strategy exists to do, would say more.

  4. Which finding is the strongest sign that a strategy is broken and not just in a bad stretch?

    1. It lost money last year
    2. It failed in exactly the conditions it was designed for
    3. Its win rate is under 50%
    Show the answer

    B. It failed in exactly the conditions it was designed for Checking the mechanism is more informative than checking the return. The trend rule exists to limit long declines, so doing almost nothing in the 2022 fall (23.8% against 24.8%) is more troubling than years of lagging in a rising market.

Exercise

One task to do in Finabase. Nothing is graded or saved.

Open IWM beside SPY on the Comparison tab. In the table, read the 1-year return and 2-year return rows for both funds, then set the chart to MAX. Is the small-company fund ahead or behind over one year, over two years, and over the whole chart? Does the answer change with the period, and would any one of those three views tell you whether the small-company premium is finished?

Compare IWM with SPY
What you should find

Whichever fund leads today, a gap over one or two years is small beside the swings in the lesson, where IWM went from 7.9 points a year ahead of SPY to 8.1 behind in different five-year blocks and ended about level over 26 years. An answer that flips with the period is the normal state. A short stretch cannot separate bad luck or absent conditions from an edge that is gone.

Sources and further reading

  1. R. David McLean and Jeffrey Pontiff, “Does Academic Research Destroy Stock Return Predictability?” Journal of Finance 71(1), 2016
  2. Campbell R. Harvey, Yan Liu and Heqing Zhu, “… and the Cross-Section of Expected Returns,” Review of Financial Studies 29(1), 2016
  3. Rolf W. Banz, “The Relationship Between Return and Market Value of Common Stocks,” Journal of Financial Economics 9(1), 1981
  4. Amir E. Khandani and Andrew W. Lo, “What Happened to the Quants in August 2007?” Journal of Investment Management 5(4), 2007
  5. SEC — Order directing the exchanges and NASD to convert to decimal pricing, Release No. 34-42914 (June 8, 2000)
  6. Finabase chart data — SPY monthly bars from February 1993 and IWM from June 2000, to October 2026; results computed by Finabase