Structured application logs and operational analytics

GraphJSON can turn structured application logs into fast operational queries, dashboards, and alerts. The useful unit is not a line of prose. It is one bounded JSON record that describes a meaningful occurrence with consistent fields.

Use this guide for application and service diagnostics. Keep distributed trace waterfalls, profiles, raw stack archives, and guaranteed paging in the specialized systems that own those workflows.

Separate logs from product events#

Product events describe durable business facts:

invoice_paid
report_published
workspace_created

Operational logs describe what a system observed while doing work:

dependency_timeout
job_retry_scheduled
request_completed

Put them in different collections when they need different owners, sensitivity, volume, or retention:

product_events
application_logs
security_audit_events

Do not let a high-volume debug stream determine the retention or cost of business events. Do not treat an application log as the authoritative billing, security, or order ledger.

Use one stable log contract#

A practical record:

{
  "event": "application_log",
  "event_id": "log_01J...",
  "schema_version": 1,
  "severity": "error",
  "service": "billing-api",
  "environment": "production",
  "release": "2026.07.26.3",
  "operation": "invoice.finalize",
  "message_template": "billing_provider_timeout",
  "error_type": "ProviderTimeout",
  "error_fingerprint": "provider_timeout:invoice_finalize:v2",
  "retryable": true,
  "duration_ms": 5032,
  "route": "POST /v1/invoices/:id/finalize",
  "status_code": 503,
  "request_id": "req_01J...",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "account_id": "acct_7"
}

Send occurrence time as the GraphJSON request timestamp. Keep insert_timestamp or observed_at only when you need to measure delivery delay.

Required fields should usually include:

Add request, trace, account, job, and provider identifiers only where they are available and safe.

Keep severity small and meaningful#

Use a controlled set:

debug
info
warn
error
fatal

Define severity by expected operational impact, not by which logging method a developer happened to call.

• debug: temporary diagnostic detail; normally disabled or heavily sampled
• info: expected lifecycle occurrence worth querying
• warn: handled degradation or recoverable unexpected behavior
• error: an operation failed or exhausted its recovery path
• fatal: the process or essential subsystem cannot continue

Do not promote every caught exception to error. A handled client cancellation or expected validation failure can be informational. Conversely, a failed business operation can deserve error even when no language exception was thrown.

If you already emit OpenTelemetry logs, map its timestamp, severity, resource, trace, span, body, and attributes into a deliberately smaller GraphJSON contract. The OpenTelemetry logs data model is a useful source vocabulary; it is not a requirement to copy every attribute.

Prefer templates over free-form messages#

This:

message_template = billing_provider_timeout
provider = stripe
operation = invoice.finalize

is easier to group than:

Stripe timed out while finalizing invoice inv_391 for Acme

Keep variable values in separate fields. A free-form message can still help with a human investigation, but:

• remove identifiers and secrets
• cap its length
• never split a chart by it
• do not depend on exact wording in an alert
• assume it can create high cardinality

Use error_type for a stable class and error_fingerprint for a reviewed grouping rule. Do not fingerprint the entire message when it includes IDs, timestamps, or random values.

Correlate without copying payloads#

Useful correlation fields:

Trace and span IDs are opaque correlation values, not user identities. Keep them as strings. If span_id is present, include trace_id too.

Do not put request bodies, response bodies, authorization headers, cookies, access tokens, card details, prompts, or complete customer records into a log merely to avoid visiting the source system.

Treat exceptions as sensitive#

Exception messages and stack traces can contain:

• file paths
• SQL values
• customer content
• URLs and query parameters
• credentials embedded in library errors

Prefer:

{
  "error_type": "ProviderTimeout",
  "error_code": "billing_provider_timeout",
  "error_fingerprint": "provider_timeout:invoice_finalize:v2"
}

If you retain a redacted stack excerpt, keep it bounded and in a short-retention collection. GraphJSON accepts at most 10,000 serialized characters per event; that limit is a guardrail, not a target payload size.

Instrument requests once#

For request analytics, emit one terminal record after the status and duration are known:

{
  "event": "request_completed",
  "severity": "info",
  "service": "reporting-api",
  "environment": "production",
  "release": "2026.07.26.3",
  "route": "GET /v1/reports/:id",
  "method": "GET",
  "status_code": 200,
  "status_family": "2xx",
  "duration_ms": 184,
  "request_id": "req_01J..."
}

Use a normalized route template, not a raw path containing customer IDs or query strings. Do not emit separate “request started” and “request succeeded” records unless the start record answers a distinct operational question.

Deliver with the right durability#

Debug and routine request logs can often tolerate loss during a process crash. Billing corrections, security audit facts, and irreversible workflow outcomes usually cannot.

Classify before building:

Batch up to 50 records per /api/bulk-log request. Monitor queue age, rejected events, and drops. Keep live product ingestion capacity available during a log spike.

Sample intentionally#

Sampling changes the denominator. Record:

• sampling rule version
• sampling probability or class
• whether the record is retained deterministically

Prefer deterministic sampling by a stable request or trace ID so all related records make the same decision.

Do not use a sampled stream to calculate exact customer usage, financial totals, security audit completeness, or per-account quotas. When sampling request logs, either preserve unsampled counters elsewhere or weight only analyses whose sampling design supports it.

Consider retaining:

• all fatal records
• all or most error records
• a reviewed fraction of successful high-volume requests
• short-lived debug records only during an investigation

Query errors by release#

SELECT
  JSONExtractString(json, 'release') AS release,
  JSONExtractString(json, 'service') AS service,
  count() AS errors
FROM application_logs
WHERE
  JSONExtractString(json, 'severity') IN ('error', 'fatal')
  AND toDateTime(timestamp) >= subtractHours(now(), 24)
GROUP BY release, service
ORDER BY errors DESC

Always compare error volume with request volume. A release serving more traffic can produce more errors while having a lower error rate.

Rank error fingerprints#

SELECT
  JSONExtractString(json, 'error_fingerprint') AS fingerprint,
  JSONExtractString(json, 'service') AS service,
  count() AS occurrences,
  uniqExact(JSONExtractString(json, 'request_id')) AS requests,
  max(toDateTime(timestamp)) AS last_seen
FROM application_logs
WHERE
  JSONExtractString(json, 'severity') IN ('error', 'fatal')
  AND JSONExtractString(json, 'error_fingerprint') != ''
  AND toDateTime(timestamp) >= subtractDays(now(), 7)
GROUP BY fingerprint, service
ORDER BY occurrences DESC
LIMIT 50

A fingerprint groups known failure classes. It does not prove that every occurrence has one root cause. Verify representative records before merging issues.

Investigate one request#

SELECT
  toDateTime(timestamp) AS occurred_at,
  JSONExtractString(json, 'service') AS service,
  JSONExtractString(json, 'operation') AS operation,
  JSONExtractString(json, 'severity') AS severity,
  JSONExtractString(json, 'message_template') AS message_template,
  JSONExtractString(json, 'error_type') AS error_type
FROM application_logs
WHERE JSONExtractString(json, 'request_id') = 'req_01J...'
ORDER BY timestamp
LIMIT 200

Run request-level queries only in an authorized operational workflow. A request ID is not permission to expose another customer’s activity.

Build the operations dashboard#

Start with:

1. request volume by service
2. 5xx rate by normalized route
3. p50 and p95 request duration
4. errors by service and release
5. top error fingerprints
6. queue age and dropped-log count from the producer
7. last observed production record

Keep ingestion health separate from service health. “No errors” can mean the service is healthy, the logger is broken, or the collection stopped receiving events.

Use GraphJSON alerts for focused threshold or comparison signals. Keep paging guarantees, acknowledgements, escalations, and on-call routing in an incident management system.

Choose retention by purpose#

Example:

request summaries: 30 days
error records: 90 days
debug investigation: 7 days
security audit facts: policy-defined authoritative system
product events: separate policy

Measure daily volume and payload size before choosing an indefinite period. High-cardinality identifiers are useful for investigation but expensive to keep forever.

Production checklist#

• Product facts, application logs, and security records have explicit boundaries.
• Severity, service, operation, route, release, and fingerprint are controlled fields.
• Variable values are separate from message templates.
• Raw bodies, headers, tokens, cookies, and customer content are excluded.
• Trace and request IDs are opaque and bounded.
• Sampling rules and analytical consequences are documented.
• Important records use a durable delivery path.
• Dashboard totals distinguish service health from ingestion health.
• Retention reflects sensitivity, volume, and investigation needs.
• Operational alerts have a separate acknowledgement and escalation owner.

Continue with Measure release impact, SLOs, error budgets, and alerts, Background jobs and queue analytics, Security audit event analytics, and Customer-support account timelines.