Regression Toward the Mean: Lessons from 35 Years of Portfolio Backtesting

In my last post, I introduced the principle of regression toward the mean — the tendency for extreme outcomes to move closer to average over time. In this article, I’ll illustrate how this principle plays out in investing, using a backtested comparison of portfolio strategies across the S&P 500 since 1990.

Backtesting the Strategies

To explore the impact of weighting methods, I simulated five portfolio strategies from 1990 through the end of 2024. Results will be updated annually to keep the analysis current.

• S&P 500 Index (Benchmark): Buy and hold the index.

• Top 20 Market-Cap Weight: Allocate capital proportionally to the 20 largest companies.

• Top 10 Equal-Weight: Invest equally across the 10 largest companies.

• Top 11–20 Equal-Weight: Invest equally across companies ranked 11–20.

• Top 20 Equal-Weight: Spread capital evenly across the 20 largest companies.

Each year, portfolios are rebalanced using the prior year’s market cap rankings. Aside from the index benchmark, all strategies involve one annual sell and one annual buy.

Data sources: FinHacker.cz for the top 20 S&P 500 constituents and Yahoo Finance for historical stock prices. You can also view the dataset and results in this GitHub repository.

A custom program was used to simulate annual rebalancing for each strategy and generate the backtesting results. Portfolios were rebalanced once per year, based on the prior year’s market-cap rankings, with one sell and one buy cycle around the year-end.

Interactive results are available in this Tableau dashboard. Or you could also view the Tableau story for essential points behind the data.

Key Observations

The backtest reveals several important patterns:

• Consistent Outperformance: All weighted strategies beat the S&P 500 benchmark over 35 years, though margins varied.

• Boom Periods Favor Top 10 Equal-Weight: During 1996–2000 and 2017–2020, this strategy led the pack, reflecting the Matthew Effect, where success compounds as capital flows increasingly toward already dominant “hot” stocks. However, after the dot-com crash, it suffered a brutal 68.9% drawdown, underscoring how concentrated momentum can reverse sharply when regression toward the mean takes hold.

• Regression at Work: Equal-weighting the top 10 and top 20 outperforms market-cap weighting most of the time, demonstrating that allocating to “not-so-hot” companies mitigates the bias toward already dominant firms.

• Top 11–20 Equal-Weight Resilience: This group outperformed during 2000–2008, when many tech giants faltered. It was also the only strategy to reach a new all-time high before the Great Recession, underscoring its relative strength in that cycle. Yet it too faced a 57.1% drawdown during the downturn.

• Long-Term Returns:

  • S&P 500 Index: +1535% (8.12% annualized)

  • Top 20 Market-Cap Weight: +2624% (9.79% annualized)

  • Top 10 Equal-Weight: +2649% (9.82% annualized)

  • Top 11–20 Equal-Weight: +2405% (9.57% annualized)

  • Top 20 Equal-Weight: +2715% (9.89% annualized)

Notably, dividends and yields were excluded, so reinvestment would push returns even higher.

The Top 10 Equal-Weight strategy led during boom periods like 1996–2000 and 2017–2021, but suffered steep declines after the dot‑com crash. In contrast, the Top 11–20 Equal-Weight strategy outperformed from 2000–2008, showing how leadership shifts as market extremes regress toward the mean.

Cycles and Predictions

Recent years echo past booms: AI-driven stocks have propelled the top 10 equal-weight strategy. But history warns us that “this time is different” is the most expensive phrase in finance. As Mark Twain quipped, “History doesn’t repeat itself, but it often rhymes.” If history rhymes again, we may see tech leaders underperform for years, much like 2000–2008.

Tech cycles also appear unusually long — nearly two decades between the dot-com boom and the next surge. If today’s rally proves another bubble, stagnation could follow.

Timing and Final Thoughts

Regression toward the mean explains why timing matters: markets swing to extremes, then regress back toward the average. Avoiding the downward regression during crashes — and re-entering for the recovery — can dramatically magnify long-term returns. For example:

• Dot-Com Bust (2000–2002, Top 10 Equal-Weight): Exiting in May 2000 and re-entering in late 2002 would have boosted returns from $2.7M to $53M, nearly doubling annualized performance.

• Great Recession (2008–2009, Top 20 Equal-Weight): Had the 2008 crash been sidestepped, $2.8M would have grown to $42M, lifting annualized returns from 9.9% to 18.8%.

• COVID Crisis (2020, Weighted Strategies): Even avoiding just one year of turmoil during the pandemic would have doubled portfolio capital, with weighted strategies reaching $4–5 million instead of approximately $2.5–2.8 million.

These scenarios show that avoiding only one major downturn can transform results — sometimes doubling both total capital and annualized returns. While timing the market is notoriously difficult, the lesson is clear: sometimes the best way to make a fortune is simply doing nothing during periods of extreme volatility.

Conclusion

Regression toward the mean is not just a statistical curiosity; it’s a force that shapes portfolios, cycles, and investor fortunes. Equal-weight strategies often outperform by countering the bias toward dominant firms, but they too are vulnerable to cycles. Over 35 years, the lesson is clear: markets reward balance, patience, and respect for history’s rhymes.

Reference materials

The Big Secret for the Small Investor: A New Route to Long-Term Investment Success. This book is a concise guide aimed at helping everyday investors outperform traditional index funds and even professional managers. In Chapter 7, he critiques market-cap-weighted indexes for concentrating too heavily on already dominant firms and highlights the advantages of equal-weight strategies, which spread risk more evenly and benefit from regression toward the mean. His evidence shows that equal-weight portfolios have historically delivered stronger long-term returns than market-cap-weighted ones, reinforcing the patterns observed in this backtest.

The Real Dogs of the Dow. This paper examines changes in the Dow Jones Industrial Average and highlights a striking phenomenon: stocks removed from the index often outperform those added as replacements. In some historical periods, the cumulative performance of a “deletion portfolio” relative to an “addition portfolio” reached ratios as high as 30 to 40 times, underscoring the potential pitfalls of relying solely on index inclusion as a signal of future success.

S&P Equal Weight vs. Cap Weight. This article analyzes the growing divergence between the market-cap-weighted S&P 500 and its equal-weighted counterpart, driven largely by the dominance of mega-cap stocks such as Apple, Microsoft, and NVIDIA. While the cap-weighted index has outperformed in recent years due to these firms’ outsized influence, the piece highlights that over longer horizons the equal-weighted index has often led, benefiting from greater diversification and regression toward the mean. Sector comparisons further show how equal-weighting reduces concentration risk, creating a more balanced portfolio structure.