When Missing Voices and Missing Winners Mislead: Rethinking Retrospective Studies

Retrospective studies are tempting: they use existing data, are cheaper than experiments, and can point to interesting patterns. But patterns are not causes. Two very common biases — survivorship bias and self-selection bias — quietly distort many retrospective findings so that results look stronger, clearer, and more actionable than they really are. Below I explain each bias with vivid examples, show how they sneak into everyday research, and offer practical ways to judge and improve retrospective claims.

Survivorship bias

Survivorship bias happens when we study only the survivors and ignore those who failed or disappeared. The result is a distorted picture that praises certain traits or choices simply because unsuccessful cases with the same traits are missing from the dataset.

Famous bomber example

During World War II analysts examined returning bombers and asked where to add armor. They mapped bullet holes on planes and found heavy damage in certain areas. The intuitive (but wrong) idea was to reinforce those damaged regions. Statistician Abraham Wald pointed out the mistake: the bullet holes were on planes that survived. The undamaged areas on returning planes were likely the places that, when hit, caused planes to be lost and never return. The correct action was to armor the spots that showed no damage on survivors.

This hypothetical pattern of damage of surviving aircraft shows locations where they can sustain damage and still return home. If the aircraft was reinforced in the most commonly hit areas, this would be a result of survivorship bias because crucial data from fatally damaged planes was being ignored; those hit in other places did not survive. Image from Wikipedia.

How survivorship bias hides in retrospective research

Company success studies often analyze only surviving firms. Bankrupt companies that shared the same characteristics are missing, making a trait look predictive when it may not be.
Longevity studies that sample long-lived people ignore others with the same lifestyles who died younger, exaggerating the connection between habits and lifespan.
Investment analyses that look at top-performing traders or strategies typically exclude those who blew up their accounts using the same methods.
Studies of “great companies” list common traits among winners but ignore many companies with the same traits that failed.
Stories about famous dropouts who became billionaires sidestep the many dropouts who struggled or remained average, implying dropout as a causal secret to success when it is often survivorship and selection telling the tale.
Startups like Airbnb or Stripe become templates for “how to build a company,” while the thousands of failed startups that followed similar playbooks are invisible. That makes certain choices (e.g., pivot timing, growth hacks) look more predictive of success than they really are.
You name it: if failed or disappeared cases aren’t in the sample, conclusions about predictors of success are unreliable.

A common pitfall across these examples is simple: we only observe the successful cases while ignoring the failed ones. That truncation of the dataset creates an upward bias in estimates of effectiveness, skill, or causal impact.

Self-selection bias

Self-selection bias arises when the people included in a dataset choose—explicitly or implicitly—to be there. When participation depends on factors that also influence the outcome, observed associations can be misleading.

UC Berkeley admissions example

A celebrated real-world illustration is the graduate admissions controversy at a major university in the 1970s(I first encounter this example from the book Standard deviations: flawed assumptions, tortured data, and other ways to lie with statistics). At the aggregate level the data initially suggested women were being admitted at lower rates than men, which looked like discrimination. But when admissions were broken down by department, many departments actually had higher acceptance rates for women; the aggregate disparity emerged because women tended to apply to more competitive departments with lower admission rates. This is a manifestation of Simpson’s paradox, and it reveals the confounding role of self-selection: applicants’ choices about where to apply are correlated with both gender and admission probability, so simple aggregation hides the true structure of the problem as shown in the following excel sheet.

Other everyday examples

Online surveys that depend on voluntary participation primarily capture people who want to express an opinion online, leaving out quieter or less-engaged populations.
Nutritional studies that associate fish consumption with longevity (as study published here) may reflect that people who eat more fish are wealthier(fish is generally more expensive than meat anyway), more health-conscious, or have better access to healthcare. These confounding lifestyle and socioeconomic factors, which influence both fish intake and lifespan, make direct causal claims precarious unless they’re controlled for properly.
Many reports present average starting-salary gaps as if attending university caused the entire difference. But self‑selection is likely at work: students who go to university often come from higher socioeconomic backgrounds, have stronger school networks, access to better career guidance, or greater family resources. They may also be more career‑oriented or risk‑tolerant in ways that affect salary outcomes. Those demographic and motivational differences — not just the credential itself — can drive much of the observed gap. Without accounting for these selection pathways, attributing the salary difference purely to university attendance overstates the causal role of the degree.

“People who make different choices may in fact be different.” — Gary Smith, Standard Deviations

Self‑selection makes comparison groups systematically different in ways driven by hidden confounding factors, so an apparent effect may disappear or even reverse once you examine disaggregated data by department, as the 1973 UC Berkeley admissions case famously demonstrated.

Root causes and why controlled experiments matter

Why are these biases so common? Because observational retrospective studies are often the only feasible option for many sociological, economic, or health questions. People, firms, and countries are not randomly assigned to policies, diets, or life events in the real world, so observational datasets naturally reflect choices, survivals, and exits that carry causal information with them.

Here are the key points:

The gold standard for causal inference is a well-designed randomized controlled trial where treatment is randomly assigned and control conditions are observed in parallel. That removes both survivorship truncation and self-selection at the assignment stage.
For many large-scale social questions, randomized trials are expensive, unethical, or unfeasible. Researchers therefore rely on observational methods and try to adjust for confounders, but adjustments depend on observing and measuring all relevant confounding factors. Unobserved confounders and missing failed cases break those adjustments.
Without randomized assignment — or clever quasi-experimental designs that convincingly mimic it — we cannot reliably separate cause from correlation. A claim might be true, partially true, or entirely spurious; retrospective data alone often cannot tell us which.

Even randomized experiments need replication. Single trials can produce results driven by chance, context, or implementation differences. Repeatable experiments reduce the chance that one lucky or unlucky study leads us to the wrong policy or belief; see the Related posts section for a deeper discussion of this topic.

How to read retrospective studies critically

Ask whether the dataset excludes failures or missing cases. If so, treat claims about “what leads to success” with extra skepticism.
Look for evidence of selection pathways. Who chose into the sample? Why? How might those choices correlate with the outcome?
Check whether researchers control for plausible confounders and whether those confounders are measured adequately.
Favor studies that use natural experiments, instrumental variables, matching, difference-in-differences, regression discontinuity, or randomized assignment when available.
Watch for claims that sound causal but are based on cross-sectional correlations or on comparisons of winners only.
Prefer replication and meta-analyses over single-study claims. Robust findings should persist across contexts, methods, and datasets.

Conclusion

Retrospective studies are useful for spotting patterns and generating hypotheses, but they are often less persuasive than they first appear because survivorship and self-selection bias lurk everywhere. Be skeptical of neat success stories, simple causal claims from observational data, and headlines that treat correlation as causation. Ask how the sample was formed, what was left out, and whether alternative explanations could produce the same pattern. If you read a counterintuitive or headline-grabbing result, dig into the methods before you take it as prescriptive advice.

I’d love to hear which examples you’ve seen that seemed convincing until you noticed one of these biases. Share a study or article in the comments and we’ll analyze where survivorship or self-selection might be shaping the story.

These related posts explore how luck, randomness can create the illusion of causal patterns in social science and finance.

Search This Blog

Complexity Thinking