The reality is you can find “evidence” for almost any narrative. Limit the sample size, cherry-pick studies, etc. Systematic reviews, meta analyses, and randomized controlled trials are all susceptible to selective interpretation/narrative fallacy.
@nntaleb 1/10
Science is assumed to be “evidence-based” but that term alone doesn’t mean much. What constitutes good evidence? How is evidence being used? Is it supporting or refuting a hypothesis? Was the hypothesis and experimental design predetermined or found ex post facto?
The reality is you can find “evidence” for almost any narrative. Limit the sample size, cherry-pick studies, etc. Systematic reviews, meta analyses, and randomized controlled trials are all susceptible to selective interpretation/narrative fallacy.
At the heart of the problem is the over-reliance on simplistic statistical techniques that do little more than quantify 2 things moving together.
Take Pearson’s correlation, based on covariance. Variation can increase simultaneously across 2 variables for countless reasons, most of which are spurious. Yet this simple notion of “causality” undergirds much of scientific literature.
Information-theoretic (entropy based) approaches on the other hand can assess *general* measures of dependence. Rather than some specialized (linear) view based on concurrent variation, entropy encompasses the amount of information contained in and between variables.
If you were genuinely interested in giving the term “evidence” an authentic and reliable meaning then the methods used to underpin an assertion would be rigorous.
We wouldn’t look to conveniently simplistic methods to denote something as evidential, rather we would look for a measure capable of assessing the expected amount of information held in a random variable; there is nothing more fundamental than information.
Consider Mutual Information (MI), which quantifies the amount of information obtained about one random variable through observing another random variable. This observing of the relationship between variables is what measurement and evidence is all about.
MI determines how different joint entropy is from marginal entropies. If there is a genuine dependence between variables we would expect information gathered from all variables at once (joint) to be less than the sum of information from independent variables (marginals).
If “evidence-based” science was genuinely invested in authentic measurement it would leverage *general* measures of dependence; that demands an approach rooted in information-theory. Without entropy you’re just picking data, choosing a narrative, and calling it “evidence.”
More from Science
@mugecevik is an excellent scientist and a responsible professional. She likely read the paper more carefully than most. She grasped some of its strengths and weaknesses that are not apparent from a cursory glance. Below, I will mention a few points some may have missed.
1/
The paper does NOT evaluate the effect of school closures. Instead it conflates all ‘educational settings' into a single category, which includes universities.
2/
The paper primarily evaluates data from March and April 2020. The article is not particularly clear about this limitation, but the information can be found in the hefty supplementary material.
3/
The authors applied four different regression methods (some fancier than others) to the same data. The outcomes of the different regression models are correlated (enough to reach statistical significance), but they vary a lot. (heat map on the right below).
4/
The effect of individual interventions is extremely difficult to disentangle as the authors stress themselves. There is a very large number of interventions considered and the model was run on 49 countries and 26 US States (and not >200 countries).
5/
1/
I've recently come across a disinformation around evidence relating to school closures and community transmission that's been platformed prominently. This arises from flawed understanding of the data that underlies this evidence, and the methodologies used in these studies. pic.twitter.com/VM7cVKghgj
— Deepti Gurdasani (@dgurdasani1) February 1, 2021
The paper does NOT evaluate the effect of school closures. Instead it conflates all ‘educational settings' into a single category, which includes universities.
2/
The paper primarily evaluates data from March and April 2020. The article is not particularly clear about this limitation, but the information can be found in the hefty supplementary material.
3/
The authors applied four different regression methods (some fancier than others) to the same data. The outcomes of the different regression models are correlated (enough to reach statistical significance), but they vary a lot. (heat map on the right below).
4/
The effect of individual interventions is extremely difficult to disentangle as the authors stress themselves. There is a very large number of interventions considered and the model was run on 49 countries and 26 US States (and not >200 countries).
5/
#DDDD $LBPS $LOAC
A thread on the potential near term catalysts behind why I have increased my position in 4d Pharma @4dpharmaplc (LON: #DDDD):
1) NASDAQ listing. This is the most obvious.
The idea behind this is that the huge pool of capital and institutional interest in the NASDAQ will enable a higher per-share valuation for #DDDD than was achievable in the UK.
Comparators to @4dpharmaplc #DDDD (market capitalisation £150m) on the NASDAQ and their market capitalisation:
Seres Therapeutics: $2.33bn = £1.72bn (has had a successful phase 3 C. difficile trial); from my previous research (below) the chance of #DDDD achieving this at least once is at least
Kaleido Biosciences: $347m = £256m. 4 products under consideration, compared to #DDDD's potential 16. When you view @4dpharmaplc's 1000+ patents and AI-driven MicroRx platform (not to mention their end-to-end manufacturing capability), 4d's undervaluation is clear.
A thread on the potential near term catalysts behind why I have increased my position in 4d Pharma @4dpharmaplc (LON: #DDDD):
1) NASDAQ listing. This is the most obvious.
The idea behind this is that the huge pool of capital and institutional interest in the NASDAQ will enable a higher per-share valuation for #DDDD than was achievable in the UK.
Comparators to @4dpharmaplc #DDDD (market capitalisation £150m) on the NASDAQ and their market capitalisation:
Seres Therapeutics: $2.33bn = £1.72bn (has had a successful phase 3 C. difficile trial); from my previous research (below) the chance of #DDDD achieving this at least once is at least
While looking at speculative pharmaceutical stocks I am reminded of why I am averse to these risky picks.#DDDD was compelling enough, though, to break this rule. The 10+ treatments under trial, industry-leading IP portfolio, and comparable undervaluation are inescapable.
— Shrey Srivastava (@BlogShrey) December 16, 2020
Kaleido Biosciences: $347m = £256m. 4 products under consideration, compared to #DDDD's potential 16. When you view @4dpharmaplc's 1000+ patents and AI-driven MicroRx platform (not to mention their end-to-end manufacturing capability), 4d's undervaluation is clear.
