False Positives, FWER, and FDR Explained

If you torture your data long enough, they will tell you whatever you want to hear - Mills (1993) False positives via statistical hypothesis testing are a severe problem in the scientific literature (Ioannidis, 2005). If a statistically significant finding looks real, but it’s not, and we make policy or clinical decisions based on this finding, it can have catastrophic consequences. Unfortunately, many researchers are still unaware exactly why false positives are so prevalent in the scientific literature, so, I’ve decided to explain some of the common reasons for the high prevalence. Read More

Meta-Analysis: Choose Your Model Wisely

Why Do a Meta-Analysis? Meta-Analyses are statistical techniques where we take the treatment effects from several studies and pool them to estimate the overall treatment effect. This type of analysis is conducted so that we can get a precise estimate of the true treatment effect or estimate the mean of several true treatment effects and to notice how robust the effects are across studies. If the individual study treatment effects tend to vary substantially across the studies (true heterogeneity, separated from random error), then we want to explore the sources of this observed dispersion. Read More

Myth: Covariates Need to Be Balanced in RCTs

Before I get into the nitty-gritty, I want to remind everyone why we randomize in the first place. It’s to reduce selection bias and to get rid of systematic variation amongst groups, which allows us to come to more precise and efficient causal inferences. Many critics claim that we can’t make valid causal inferences if there’s an imbalance in covariates between the groups. Here’s an example. Say, hypothetically, we had two groups, and we wanted to see the effect of a statin on all-cause mortality and compare it to placebo. Read More

Four Misconceptions About Statistical Power

Statistical power, within the context of hypothesis testing, is the probability of rejecting the test hypothesis at a specified level, given that the alternative hypothesis is true. In simpler words, it’s the probability of finding a statistically significant effect, when there is one. Here, I address some misconceptions I often see about statistical power. Misconception: Statistical power can only be increased with larger sample sizes Truth: Nope. Here are some other things that increase statistical power: Read More