Google SRE Prodcast on YouTube · Open on YouTube
Running production: reliability, observability, performance and security · Proficient · Lesson 72 of 98 · 40 min video, then about 7 hours of practice
Being on call and responding to incidents
After this you can respond to a production alert calmly: assess impact, mitigate first, communicate status and hand over cleanly.
Why it matters
On call is where you learn how your system really behaves. The first rule of an incident is mitigate, then diagnose: roll back, fail over, or turn off the feature, and find the root cause once users are no longer affected. Communicate early and regularly, even when there is nothing new. Write down what you do as you do it; the timeline is gold for the postmortem. Trap: debugging the root cause while users are still affected. Stop the bleeding first.
How to use this resource
Google's SRE team on how incident management actually works.
Go further
Go deeper