minutephysics on YouTube · Open on YouTube
Experimentation, causal thinking and metric design · Proficient · Lesson 59 of 99 · 5 min video, then about 1 hour of practice
Confounding, selection bias and Simpson's paradox
After this you can spot confounders and selection effects in an observational comparison, and show how aggregation can reverse a trend.
Why it matters
Most questions analysts get cannot be randomised: 'do users who use feature X retain better?'. Usually they do, because engaged users use more features. Learn to name the confounder, recognise selection bias (comparing people who chose something with people who did not), and check whether a pattern reverses within segments. Saying clearly what an observational comparison cannot prove is a senior skill.
How to use this resource
A five-minute visual explanation of Simpson's paradox and why aggregation can reverse a conclusion.