the new tabletop simulator update.... oh no

Survivorship Bias 101

History is written by the victors. But if we exclusively focus on these successes, we allow survivorship bias to distort our understanding of the world.

But what is "survivorship bias" and how does it work?

Here's Survivorship Bias 101!

👇👇👇

1/ First, a few definitions.

Survivorship bias is the logical error of concentrating on survivors (successes) and ignoring casualties (failures).

When we do so, we miss the true "base rates" of survival (the actual probability of success) and arrive at flawed conclusions.

2/ When we completely ignore failures, we lose our ability to correctly identify the differences between successes and failures.

Put simply, exclusively focusing on successes may actually inhibit our ability to identify (and replicate) the actions that led to such success.

3/ Cicero wrote on the topic over 2,000 years ago.

An atheist named Diagoras is shown portraits of people who prayed and were saved from death at sea as proof of God's existence.

Diagoras replies, "I see those who were saved, but where are those painted who were shipwrecked?"

4/ Cicero cuts right to the point - we cannot conclude that prayer led to being saved from the sea if we ignore those who prayed and then drowned.

Survivorship bias can lead to a deeply-flawed understanding of cause-and-effect relationships.

Let's look at some examples.

There is a highly instructive symposium on "Veil of Ignorance" #ProcessTracing in the new @APSAtweets QMMR publication https://t.co/nxqEQ9HaRz There is lots to say ab it. For now, I want to focus on whether (causal) qualitative research has a problem w/ confirmation bias that 1/

needs to be addressed.
In short, Veil of Ignorance process tracing aims to blindfold the data collectors and analysts to the hypotheses to reduce risk of confirmation bias.
Proposal receives a lot of criticism by all respondents, partly for valid reasons, partly not, IMO 2/

If there was no bigger issue with confirmation bias, there would not be much benefit in talking about blinding in qualitative research.
As Beach notes, the intro to the symposium does not present evidence that bias is a problem. This is correct, but there is also no evidence 3/

that confirmation bias is not a problem. It's an absence of evidence situation with regard to comprehensive, direct evidence, i.e. from verification studies of qualitative research (for studies that are built on an epistemology for which the idea of verification makes sense). 4/

However, I'd say there is circumstantial evidence that confirmation bias is a problem.
1) Evidence for confirmation bias in quantitative research: The developments around #OpenScience and transparency in quantitative research in different disciplines present diverse evidence 5/