Survivorship Bias: How a cognitive bias prevents from seeing the full picture

Imagine a doctor prescribing treatment based only on patients who recovered and ignoring those who failed. The resulting picture of the disease would be inaccurate. This is “survivorship bias” — a cognitive bias that causes people to overestimate successes and underestimate the true extent of problems, creating a dangerous illusion of control.

Survivorship bias is a cognitive misrepresentation in which attention is focused only on the “surviving” (successful) elements of a sample rather than on the entire data set. Like many other cognitive biases, particularly confirmation bias, it is an unconscious, systematic error in thinking that occurs when processing and interpreting information. This type of error is also regularly encountered by those tasked with verifying information.

The most iconic example of survivorship bias comes from World War II сase. The most intact aircraft returned with the most holes in their wings and tail, so it was these areas that were initially considered for armor plating. However, Abraham Wald noted that the holes indicated areas that could withstand hits, and when shells struck more vulnerable areas, the aircraft no longer returned. Therefore, to improve the resilience of the entire bomber fleet, armor should have been reinforced in areas where there were almost no holes — in the engine compartment and around the cockpit. Taking this factor into account significantly reduced losses during sorties.

In fact-checking, survivorship bias occurs when analysis focuses only on “survivors” or successful cases, omitting those that “failed” or failed to achieve significant results. By focusing solely on successful debunkings, fact-checkers risk overlooking “hidden” or undetected instances of disinformation.

Factors that influence the occurrence of survivorship bias

• Lack of a complete information picture.

Data on unsuccessful cases may be unavailable or closed due to their sensitivity or lack of reporting.

One of the key areas where this factor manifests itself is in the medical field. During the COVID-19 epidemic in some countries, due to overburdened infrastructure or deliberate concealment, mortality data, as researchers note, may have been irrelevant. At the same time, the mortality rate is one of the most important indicators of the danger of a disease, the failure to take into account of which can have inevitable consequences.

A similar problem is seen in other medical studies. For example, when studying life-threatening diseases such as cancer, there is a risk that data from the sickest patients, which could have a significant impact on the results, is often lost: patients may not survive to the end of the study, withdraw from the study, or be unable to provide the necessary information due to deteriorating condition.

In fact-checking, this factor manifests itself in a distorted assessment of the scale of disinformation. When fact-checkers focus primarily on public cases that have already gained significant traction, complex, targeted, or low-profile disinformation campaigns that have not achieved widespread distribution (and therefore have not been verified) remain under the radar. This hinders the development of strategies to counter more sophisticated manipulation.

Thus, a distorted “threat map” is created, creating the illusion that the main problems are fake news that have “taken off.” Meanwhile, quiet but persistent false narratives within narrow communities can cause significant long-term damage.

• Selectivity in the dissemination of information in the information field.

The media, authors, and, to some extent, researchers in a significant number of cases focus on positive rather than negative examples, looking primarily at success stories rather than failures.

Exposing success stories of people without a college degree distorts the perception of successful strategies. An experiment confirmed this: after viewing case studies of such founders, subjects were 55% more likely to bet on a company whose founder had dropped out. This creates a “survivorship bias,” where numerous unsuccessful examples of such startups are overlooked.

This problem is present, including in the scientific community. Some researchers, particularly in the medical field, have not published their work if they receive “negative” results. At the same time, even a research result that does not meet expectations is an important set of data, the presence of which can set a new direction for subsequent research.

This cognitive bias, one way or another, leads to a false perception of facts, making them difficult to verify. Thus, selective information selection in fact-checking often leads to an unintentional preference for public and easily accessible materials, which can easily lead to missing important but hidden information. This sometimes manifests itself in an excessive reliance on official sources and hinders the establishment of the truth.

It’s also worth noting that in the context of fact-checking research, there’s a risk of methodological “stagnation”: successful examples of using certain tools lead one to believe they’re reliable for all situations. Meanwhile, many cases require unconventional solutions that don’t fit within the usual set of practices.

Mechanisms of occurrence of survivor’s error

• Selective clipping

Researchers, journalists, and fact-checkers select a sample of cases during their analysis, which is often limited in some way due to data overload. Under these conditions, irrelevant cases is being clipped, as deemed by the study author, are eliminated, which can, in particular, lead to unsuccessful cases being overlooked. For example, a researcher might focus on existing companies or living individuals, which would exclude data on those who were less adaptable. This, in turn, impacts the representativeness of the sample, making it less valid.

Thus in the investment field, focusing on studying the returns of successful companies or entire sectors leads to survivorship bias. On the one hand, this approach ignores the experience of failed startups, and on the other, it can inflate sector returns by only including “surviving” successful players in the sample. As a result, investment attractiveness is assessed incorrectly.

• Neglect of probabilistic effects

Random fluctuations in data (probability dispersion) can be mistaken for a pattern. When success occurs without any connection to the factors being studied, and the researcher ignores the role of chance and fails to account for luck and statistical dispersion, they may mistakenly attribute this success to specific characteristics or actions, creating the illusion of controlled causality.

This mechanism is well illustrated by the desire to study successful startups. However, they often do not receive the development they deserve, even if they have a good idea and initial success in attracting capital. Meanwhile, companies that “thrive” are those that, among other factors, experience favorable circumstances such as market changes, the absence of key employee turnover, and so on.

Concomitant cognitive biases

• Confirmation bias. This cognitive distortion can contribute to the elimination of unsuccessful cases that do not fit into the researcher’s subjective vision and can distort the desired results of the study.

• Generalization. This thinking error, where a person makes general but false conclusions based on limited data, may also influence to incorrect interpretation in the context under consideration, since the data with a predominance of successful cases are generalized to the entire sample.

Countermeasures

To avoid survivorship bias, it is important to consciously strive to build a comprehensive database by purposefully collecting and analyzing information about unsuccessful or incomplete cases.

In deep analytical research, particularly in medicine, there are methods for accounting for censored, missing data, ensuring their greater validity.

To overcome survivorship bias when verifying data, it is important to take its presence into account, in particular, to organize a search for primary documents and “gray” sources, including unsuccessful, indirect, or unofficial evidence.

Monitoring also requires periodically randomly selecting content for review, without regard to its popularity or prior assessment of its credibility.

Furthermore, in the context of research aimed at improving the effectiveness of fact-checking, it is necessary to take into account failed checks, “false alarms,” ​​uncommon disinformation tactics, etc.

Combating disinformation requires challenging one’s own cognitive biases. Recognizing “survivorship bias” is the first step to more honest and effective fact-checking. Only by purposefully searching for “non-surviving” cases — quiet narratives, failed checks, and closed data — can create a real threat map. The effectiveness of fact-checking is determined not only by the number of exposed fake news, but also by the ability to see the entire information field, with all its dark spots.