What Is It?
Backtesting is the attempt to estimate how a trading strategy would have behaved historically. It is necessary, but dangerously seductive. A strong backtest can reflect real edge, or it can reflect data leakage, survivorship bias, overfitting, and luck.
Regime shift is the deeper problem. Even a clean backtest describes a world that may no longer exist.
How It Actually Works
A robust backtest starts with accurate timestamps, realistic costs, delisted assets, corporate actions, and no access to future information. Then it tests the full pipeline, signal generation, sizing, constraints, and execution, across multiple regimes rather than one golden period.
Overfitting happens when you tune the strategy to the quirks of historical noise. The classic symptoms are too many parameters, too much search, and fragile sensitivity to small changes in assumptions. López de Prado’s practical warning applies here: every research decision effectively spends statistical degrees of freedom, even if you do not call it a parameter.
Regime shift means the generating process changes. A strategy built for zero-rate liquidity regimes may die in a tightening cycle. A crypto microstructure edge may vanish when exchange incentives or fee tiers change. The right question is not “did it work?” but “why did it work, and what could plausibly break that mechanism?”
The Jargon Decoded
- Backtest — Historical simulation of a strategy.
- Look-ahead bias — Using information not available at the decision time.
- Survivorship bias — Ignoring failed or delisted assets.
- Overfitting — Learning noise rather than signal.
- Walk-forward test — Repeated out-of-sample evaluation through time.
- Regime shift — Structural change in the market environment.
Why This Matters
This is the difference between research and self-deception. For a technical curriculum, it is one of the highest-leverage concepts because it teaches skepticism toward pretty performance charts.
What This Unlocks
You can design more honest tests, recognize fragile edges earlier, and treat live trading as a continuation of research rather than a victory lap after backtesting.
What Still Breaks
No backtest captures adversarial adaptation perfectly. The moment a strategy becomes capitalized, its own market impact changes the environment. History can tell you a lot, but not how the future will react to your presence.
Sources
- Advances in Financial Machine Learning — Marcos López de Prado — Practical guidance on features, backtests, leakage, and overfitting.
- 151 Trading Strategies — Zura Kakushadze and Juan Andrés Serur — Broad taxonomy of strategies across equities, futures, options, FX, fixed income, and more.
- The Statistics of Sharpe Ratios — Lo — Why risk-adjusted metrics need careful interpretation under real-world assumptions.