Lenny's Podcast on YouTube · Open on YouTube
Senior product design: owning outcomes · Proficient · Lesson 63 of 99 · 1 h 23 min video, then about 2 hours of practice
Experimentation: hypotheses, A/B tests and when not to run them
After this you can write a testable design hypothesis, help design an A/B test with a primary metric and guardrails, and read a result without the common statistical errors.
Why it matters
On product teams at scale, many design decisions are settled by controlled experiments, and designers who cannot speak that language get overruled by people who can. You do not need to run the statistics, but you need to know what makes a test trustworthy: a hypothesis written in advance, one primary metric, guardrails, enough traffic to detect a realistic effect, running for full weeks, and not peeking. Equally important is knowing when an experiment is the wrong tool: low traffic, big redesigns, or questions about *why*, which need research instead. Trap: running a test on a small product and treating noise as a result.
How to use this resource
A long interview with the researcher who built experimentation platforms at Microsoft, Amazon and Airbnb and co-wrote the standard textbook on the subject. It is long because it is thorough: on trustworthiness, surprising failure rates, and the errors teams make. Best listened to as a podcast.