freeCodeCamp.org on YouTube · Open on YouTube
Experimentation, causal thinking and metric design · Proficient · Lesson 56 of 99 · 2 h 33 min video, then about 1 hour of practice
Designing a trustworthy A/B test
After this you can design an online experiment: hypothesis, randomisation unit, primary and guardrail metrics, and a decision rule fixed before launch.
Why it matters
An experiment is only as good as decisions made before it starts. Fix the hypothesis, the primary metric, the randomisation unit (user, session, account) and the decision rule in writing before launch. Learn why randomising at the wrong unit invalidates the analysis, and why the primary metric must be sensitive enough to move within the run time. Kohavi's work is the reference for doing this at scale.
How to use this resource
A two-and-a-half-hour crash course in A/B testing with a full case study, from hypothesis to analysis.