Measure SLOs, error budgets, and burn rate

GraphJSON can analyze compact reliability facts and explain which releases, routes, tenants, or dependencies consumed an error budget. It should complement an authoritative monitoring and paging system, not silently replace one.

The native alert surface watches supported Single Line collection visualizations. It does not currently evaluate arbitrary saved SQL or provide a built-in multi-window, multi-burn-rate pager.

Use this architecture:

authoritative request/monitor signals
      ├── paging system → urgent incident notification
      ├── SLO evaluator → compact window measurement
      └── GraphJSON → analysis, dashboard, and supported threshold alert

This avoids relying on the product being measured as the only way to detect its failure.

Define the service and user outcome#

An SLO starts with one service boundary and one user-visible outcome:

Service: report generation API
Population: authenticated production report requests
Good: terminal response in under 2 seconds and correct
Excluded: client cancellations before execution
Window: rolling 28 days
Objective: 99.9%
Owner: reporting platform

Do not begin with “CPU below 80%.” Resource saturation is a diagnostic signal; the SLO should usually express whether users received the promised result.

Document exclusions narrowly. Removing maintenance, dependency failures, or particular customers after an incident makes the objective untrustworthy.

Emit one terminal fact per opportunity#

Recommended event:

{
  "event": "request_completed",
  "event_id": "req_01J...",
  "request_id": "req_01J...",
  "service": "reporting-api",
  "operation": "generate_report",
  "account_id": "acct_7",
  "outcome": "success",
  "duration_ms": 483,
  "slo_eligible": true,
  "slo_good": true,
  "release": "2026.07.26.1",
  "region": "us-west",
  "schema_version": 1
}

Set slo_good after the terminal outcome is known. A handled failure is still bad for an availability SLI if the user did not receive the requested result.

Use a stable event ID and deduplicate it for exact objectives. Record attempt events separately; retries must not turn one user opportunity into several denominator rows.

Calculate the SLI#

For request-based availability:

SLI = good eligible requests / all eligible requests

WITH requests AS (
  SELECT
    JSONExtractString(json, 'event_id') AS event_id,
    argMax(
      JSONExtractBool(json, 'slo_eligible'),
      timestamp
    ) AS eligible,
    argMax(
      JSONExtractBool(json, 'slo_good'),
      timestamp
    ) AS good
  FROM application_logs
  WHERE
    JSONExtractString(json, 'event') = 'request_completed'
    AND JSONExtractString(json, 'service') = 'reporting-api'
    AND timestamp >= now() - INTERVAL 28 DAY
  GROUP BY event_id
)
SELECT
  countIf(eligible) AS total,
  countIf(eligible AND good) AS good,
  if(total = 0, NULL, good / total) AS sli
FROM requests

NULL means no opportunities. It should not be rendered as either perfect reliability or zero reliability.

For latency, define good as a request below the stated threshold. A percentile is useful diagnostically, but a threshold-based good/total ratio maps directly to an error budget.

Calculate the error budget#

For objective SLO:

allowed bad fraction = 1 - SLO
observed bad fraction = bad / total
budget consumed = observed bad fraction / allowed bad fraction
budget remaining = 1 - budget consumed

At 99.9%, the allowed bad fraction is 0.1%. If 0.2% of eligible requests are bad in the window, 200% of the budget has been consumed.

Display numerator and denominator alongside the percentage. A 50% failure rate from two requests should not be interpreted like the same rate from two million.

Use calendar and rolling views correctly#

A rolling 28-day window answers “how are we performing now?” A calendar month answers “how did the contractual month perform?” They are not interchangeable.

Store timestamps in UTC and make the reporting time zone explicit. Use a settling delay when late terminal events are normal, then label the freshest period as provisional.

Do not average daily percentages. Sum good and total opportunities across the window, then divide.

Understand burn rate#

Burn rate compares recent bad-event rate with the allowed rate:

burn rate = observed bad fraction / (1 - SLO)

For a 99.9% objective:

burn rate 1 consumes budget at the planned pace
burn rate 2 consumes it twice as fast
burn rate 14.4 consumes a 30-day budget in roughly 50 hours if sustained

High burn over a short window detects sharp incidents. Lower burn over a longer window detects slow leaks. The Google SRE Workbook’s SLO alerting chapter explains the multi-window principle and its tradeoffs.

Emit evaluated windows for native alerts#

Because GraphJSON alerts do not evaluate saved SQL directly, have the authoritative SLO evaluator emit a compact result:

{
  "event": "slo_window_measured",
  "measurement_id": "reporting-api:availability:5m:1785081900",
  "service": "reporting-api",
  "slo": "availability",
  "window": "5m",
  "window_end": 1785081900,
  "good": 24940,
  "total": 25000,
  "bad_fraction": 0.0024,
  "objective": 0.999,
  "burn_rate": 2.4,
  "evaluator_version": 3
}

Then a supported collection Single Line visualization can alert on a numeric burn_rate threshold. This is a single-window safety net, not a claim of full multi-window SLO paging.

Keep the authoritative pager in the monitoring system that evaluates both windows, missing data, notification routing, deduplication, escalation, and acknowledgment.

Handle low traffic and missing data#

Low-volume services need special care:

require a minimum denominator before a percentage page
retain an absolute bad-count condition for severe failures
lengthen the analysis window without hiding immediate total outage
show the denominator in every dashboard and notification
treat a stopped evaluator as missing, not healthy

Emit an independent heartbeat or monitor “last evaluated at.” Alert on stale evaluation through a failure domain that does not depend solely on the SLO evaluator.

Google’s monitoring guidance emphasizes symptoms, actionable signals, and avoiding pages that have no required human response; see the SRE Workbook monitoring chapter.

Build an investigation dashboard#

Recommended tiles:

rolling SLI, objective, and remaining budget
good, bad, and total opportunities
short- and long-window burn rate
bad requests by operation
bad requests by release and region
latency distribution
evaluator freshness
recent event samples with request correlation

Limit high-cardinality splits. Start with service, operation, outcome, release, and region; investigate a specific request ID only after a broad signal is known.

Make notifications actionable#

Every SLO notification should contain or link to:

service and objective
affected window and denominator
current burn rate
top failing operation or region
last known release
dashboard and runbook
owner and paging policy

Avoid alerting on every application error. Page when a user outcome consumes budget quickly enough to require action. Use tickets or review workflows for slow budget consumption.

Validate the objective#

Before production:

replay a small known set of good and bad fixture events
confirm eligibility and deduplication
test the exact time boundary
verify total = 0 produces missing data
emit a safe high-burn test measurement
confirm GraphJSON notification delivery
confirm the authoritative pager and escalation separately
exercise the linked runbook

After an incident, compare SLO events with the authoritative request source. Missing terminal events can make a broken service appear healthier.

Production checklist#

Service, population, good event, window, target, exclusions, and owner are explicit.
One deduplicated terminal event represents each user opportunity.
Good and total counts remain visible next to percentages.
Empty and low-traffic behavior are defined.
Rolling and calendar windows are labeled correctly.
An authoritative monitor evaluates paging outside GraphJSON.
GraphJSON receives compact evaluated windows for analysis or supported alerts.
Evaluator freshness has an independent signal.
Notifications link to an owner, dashboard, and tested runbook.
SLO SQL and event contracts have golden fixtures.

Continue with structured application logs, configure alerts, and monitor instrumentation health.