Emma Ding on YouTube · Open on YouTube
Experimentation, causal thinking and metric design · Proficient · Lesson 58 of 99 · 11 min video, then about 3 hours of practice
Reading an experiment without fooling yourself
After this you can analyse an experiment result, check for sample ratio mismatch, and recognise peeking, novelty effects, multiple comparisons and interference.
Why it matters
Most wrong experiment conclusions come from a short list of pitfalls: stopping early when results look good (peeking), a broken randomisation shown by a sample ratio mismatch, novelty effects that fade, testing twenty metrics and reporting the one that moved, and users in different groups affecting each other. Learn to check for each one routinely, and to say 'this result is not trustworthy' when it is not.
How to use this resource
A compact tour of common A/B testing mistakes and their fixes: peeking, multiple testing, novelty and network effects.