budding

Alert on burn rate, not on errors

An error-rate threshold pages you for blips and misses slow leaks. Multi-window burn-rate alerts page for what actually threatens the SLO.

planted Aug 30, 2026 1 min read

sre observability

A 99.9% SLO gives you ~43 minutes of budget a month. Page when you're spending it fast (2% of budget in 1h and 5m), ticket when you're spending it slowly (10% in 3 days). Both windows must agree, so a single spike doesn't wake anyone.

Severity	Long window	Short window	Burn rate
page	1h	5m	14.4×
page	6h	30m	6×
ticket	3d	6h	1×

From Site Reliability Engineering: How Google Runs Production Systems; wired into Grafana in lgtm-sidecar: a resource-capped observability stack for small teams.

Links from this note

Jan 20, 2026Site Reliability Engineering: How Google Runs Production SystemsThe chapters on SLOs, alerting and toil are the spine of how I set up on-call for small teams; the rest is Google-scale context.Read Apr 10, 2026lgtm-sidecar: a resource-capped observability stack for small teamsRead