Experimentation, causal thinking and metric design · Proficient · Lesson 64 of 99 · about 2 weeks
Project: own an experiment from design to decision
After this you have owned the analysis behind a real product or business decision, from pre-registration to readout.
Why it matters
Pick an upcoming change your team is planning and offer to own the measurement. Write the pre-registration before launch and share it with the decision maker; agree the decision rule in advance. Monitor for data-quality problems (not for significance) during the run. Read it out with an effect size, a confidence interval, the SRM check and guardrails, and recommend a decision. Afterwards, write a short retrospective: what would you do differently? If randomisation is impossible, apply difference-in-differences or another quasi-experimental design instead and state its assumptions explicitly.
How to use this resource
An open-source experimentation platform's documentation of its statistical approach; a good template for what a readout should report.