Backtesting basics: biases, costs and an honest SPY test

A backtest replays a trading rule over past prices and records what it would have done: every entry, every exit, the gains, the losses and the worst stretch along the way. It is the only way to learn whether an idea had any merit before risking money on it, and it is also one of the easiest ways to fool yourself. A careless backtest almost always looks better than the strategy will ever trade.

This guide covers the mistakes that inflate backtests, look-ahead bias, survivorship bias, overfitting and ignored costs, and the habits that keep a test honest. It then runs a small, transparent backtest of a classic rule on 32 years of SPY data and reports the real numbers, including the parts that make the rule look bad. The last section explains exactly what Finabase's backtesting page does, and what it leaves out.

What a backtest needs

Every backtest has the same parts, and each one is a place where errors creep in.

1. A precise rule. "Buy when the trend turns up" is not testable. "Buy at the next open when the monthly close is above its 10-month simple moving average; sell at the next open when it closes below" is.
2. Clean data. Prices adjusted for splits, dividends handled consistently, and the full universe of instruments that existed at each date.
3. Realistic execution. When exactly could you have traded, at what price, and at what cost?
4. Honest measurement. Return, risk and the path between them, compared with a sensible Benchmark, usually buy and hold.

The four biases that inflate backtests

Look-ahead bias

Look-ahead bias is using information that was not available when the decision was made. It is usually accidental:

• Deciding on a day's close and assuming you also bought at that close. The close is only known once trading has ended. A market-on-close order can come close, but the strict version is to trade at the next bar's open.
• Using revised data, such as restated earnings or revised economic figures, as if it had been known on the original date.
• Choosing a stock universe or a parameter with knowledge of the whole period, for example testing "stocks in today's S&P 500."

Survivorship bias

A database that holds only companies that still exist has quietly removed the bankruptcies, the acquisitions at depressed prices and the delisted failures. A rule that buys beaten-down stocks looks brilliant on survivors because the ones that went to zero are not there to be bought. Testing on index funds such as SPY, as below, avoids company-level survivorship, but not a subtler version: choosing the US stock market, one of the best-performing markets of the last century, is itself picking a survivor.

Overfitting and data snooping

Overfitting means tuning a rule until it matches the noise in the past. Each extra parameter, each indicator tried and dropped, and each tweak after looking at the results adds a chance of fitting luck. Bailey, Borwein, López de Prado and Zhu showed in 2014 that after enough trials an impressive backtest is almost guaranteed even when no strategy has any edge. Effects that are real tend to shrink too: McLean and Pontiff found in 2016 that published stock-return predictors earned noticeably less after their sample periods ended, and less again after publication. The page on the Limitations of technical analysis: what the research really shows shows the effect with 1,000 random rules.

Ignoring costs and frictions

Commissions are near zero at most US brokers in 2026, but trading is not free. Every trade crosses the bid-ask spread, suffers some Slippage, and in a taxable account may realize a taxable gain. A rule that trades weekly and earns 0.2% per trade before costs can easily lose money after them. Charge costs on every switch, and test what happens if they double.

A worked example: the 10-month moving average on SPY

The rule tested here is simple and well known. Mebane Faber popularized a version of it in 2007: hold the market when its monthly close is above its 10-month simple moving average, otherwise hold cash. Ten months is roughly the 200-day average many traders use on daily charts.

The results

The rule did what trend-following rules are known for: it sidestepped the worst of the big bear markets. In 2008 it finished roughly flat (+0.1%) while buy and hold lost 36.8%. From 2000 to 2002 it lost about 7% in total while the market fell about 38%. That is why its worst drawdown was less than half as deep.

It paid for that protection, and the bill was large:

• It lagged in rebounds and choppy years. In 2019 it made 5.9% against 31.2%, after selling during the late-2018 decline and buying back higher. In 2023 it made 9.6% against 26.2%. In 2010 it made 1.9% against 15.1%.
• It can lose more than the market. In 2022 the rule lost 22.1% while buy and hold lost 18.2%. It was whipsawed: out after a drop, back in on a rally that failed, out again lower. Its own worst drawdown, −23.5%, ran from December 2021 to February 2023.
• Over the full period it ended with about 55% of buy-and-hold's wealth. Two points of annual return compound into a large gap over three decades.

Costs mattered less than you might expect, because the rule trades only about one and a half times a year: without the 0.10% charge its annual return rises only from 8.9% to 9.1%. The bigger omission is cash yield. Treasury bills paid a little over 2% a year on average over this period, with long stretches near zero. Earning that in the 24% of months spent in cash would have added roughly half a percentage point a year: not enough to close the gap.

In sample versus out of sample

Why 10 months? To see how much that choice matters, we split the data. We tried every look-back from 2 to 24 months on March 1995 to December 2009, the in-sample period, and then ran the same 23 rules on January 2010 to October 2026, data the "designer" never looked at.

The in-sample period contained two bear markets of roughly 50% or more, perfect conditions for a trend filter. Look-backs from 7 to 24 months all beat buy and hold there, and the best, 12 months, earned 12.0% a year against 7.6%, with a worst drawdown of only −15.3%. A researcher who stopped at that point would have published a rule that added 4.4 points a year.

Out of sample, from 2010, the same 12-month rule earned 8.3% a year against 14.1% for buy and hold, trailing by almost 6 points. It ranked 16th of the 23 look-backs. Every look-back lagged buy and hold in that period, and the drawdown benefit nearly disappeared: the 12-month rule's worst month-end drawdown was −20.3%, against −23.9% for simply holding.

The lesson is not that trend filters never work. In markets with long, deep bear phases they have historically reduced the worst losses, and the 2000s were such a period. The lesson is that the best in-sample parameter is mostly a description of the in-sample period. The out-of-sample record is the honest estimate, and it is usually worse.

A checklist for an honest backtest

• Write the rule down before you test it, including every parameter, and keep a count of every variation you try.
• Trade at prices you could have gotten: next bar's open, or the close only if you can genuinely use a market-on-close order.
• Charge costs and test them doubled. Include taxes if the money is in a taxable account.
• Use the full universe, including delisted stocks, or test on instruments that have no survivorship problem.
• Hold out data. Design on one period, test once on another. Walk-forward testing repeats this: re-fit on a rolling window, trade the next window, and stitch the out-of-sample pieces together.
• Look at the whole distribution: the worst Drawdown, the longest time underwater, the worst year, the number of trades, and risk-adjusted measures such as the Sharpe ratio. A strategy with 12 trades proves nothing.
• Compare with a simple benchmark. If buy and hold beats the rule on both return and drawdown, the rule is not worth the effort.
• Paper trade before real money. A trading journal that compares live fills with the backtest catches execution assumptions that were too generous.

Measures that matter

How backtesting works in Finabase

Finabase has two related tools, both shared with the Stock Master app.

Backtesting tests indicator and candlestick signals on one symbol. You add any mix of indicators, overlays and candlestick patterns. The sample shown to visitors who are not signed in runs MACD, RSI and Parabolic SAR signals on Apple, long only, over one year. Signed in, you can save your own backtests and change the symbol, the range (from one day to the full history) and the action type: Long & short, Long only or Short only. The simulation follows the original app's rules exactly:

• Every bullish signal (an indicator crossover such as RSI rising out of oversold, or a bullish candlestick pattern) buys $10,000 of the stock; every bearish signal sells or shorts $10,000.
• A bullish signal first closes any open short, and a bearish signal first closes any open long. Repeated signals in the same direction add another $10,000 each.
• Trades fill at the close of the bar that produced the signal. Commissions are set to zero and there is no allowance for spread or slippage.
• Anything still open at the end is closed at the latest price.

The report shows the total gain, the gain as a percentage of the most capital that was tied up ("Gain / max cost"), the number of trades and the win rate of closed trades, a chart of trades on the price, a long/short/total results table, the account's market value against cost over the range, and every transaction with its signal.

Read those results with this page in mind. Filling at the signal bar's close is slightly optimistic, because the signal is only known once that close prints. Zero costs flatter rules that trade often. And with a one- or two-year range, most backtests contain only a handful of trades. Treat a good result as a hypothesis to test on other symbols and other periods, not as evidence.

Strategies handles the kind of rule tested above: if-then conditions such as "if the 50-day SMA is above the price, hold SPY, otherwise hold cash," replayed over the last two years and charted as a portfolio. Two years is a short window for a timing rule, so treat the result as an illustration of how the rule behaves, not a measure of its long-run value.

Frequently asked questions

How much data do I need for a backtest?

Enough to include different market conditions, and enough trades to matter. For a rule that trades a few times a year, that means decades and ideally several markets. For a short-term rule, a few years can contain hundreds of trades, but it may still cover only one kind of market.

What is the difference between in-sample and out-of-sample?

In-sample data is what you use to design and tune the rule. Out-of-sample data is held back and used once, after the design is frozen. Only the out-of-sample result says much about the future, because the in-sample result has been shaped by your choices.

Why does my live trading do worse than my backtest?

Usually because of costs and fills the backtest ignored, look-ahead in the test, or overfitting. Live results are a true out-of-sample test, and most strategies degrade when they meet it.

Is a moving-average timing rule a good strategy?

This page cannot answer that for you, and the example above shows why the answer is mixed. Over 32 years on SPY, the 10-month rule roughly halved the worst month-end drawdown but gave up about two percentage points of annual return, and from 2010 to 2026 every look-back we tested trailed buy and hold.

Summary

A backtest is an experiment, and like any experiment it is easy to bias. Use only information available at the time, include the losers, charge realistic costs, and hold back data you never look at while designing. Then judge the rule on that out-of-sample record, its drawdowns and its number of trades, not on its best curve. The SPY example shows both sides: a simple trend rule that sharply reduced bear-market losses, and a tuned version whose in-sample edge reversed once it met new data.