DataInterview on YouTube · Open on YouTube
Experimentation, causal thinking and metric design · Proficient · Lesson 57 of 99 · 22 min video, then about 2 hours of practice
Sample size, power and minimum detectable effect
After this you can calculate the sample size and run length an experiment needs, and explain why an underpowered test is worse than no test.
Why it matters
Most experiments in most companies are underpowered: they could not have detected a realistic effect, so a 'no significant difference' result means nothing. Learn the relationship between baseline rate, minimum detectable effect, power and sample size, and use it to tell stakeholders up front how long a test must run. The trap is choosing the minimum detectable effect after the fact to make the test look adequate.
How to use this resource
A former Google data scientist walks through sample-size calculation for A/B tests step by step.