Incident Retrospective Agent
An AI agent that reads a published outage timeline and drafts a structured retrospective: timeline, contributing factors with evidence, impact and action items. An automated eval scores it against each company's own root cause analysis.
- 4-step agent with a review step; code applies the reviewer's verdicts
- LLM-as-judge eval, 5 runs per incident, recall and precision computed in code
- Found and fixed scoring flaws by reading the agent's actual output
Results on 5 real outages (mean of 5 runs)
| Incident | Recall | Precision |
|---|---|---|
| GitLab database outage, 2017 | 100% | 100% |
| AWS S3 us-east-1, 2017 | 96% | 100% |
| Google Cloud Service Control, 2025 | 100% | 100% |
| AWS DynamoDB us-east-1, 2025 | 97% | 100% |
| CrowdStrike Channel File 291, 2024 | 92% | 100% |