Monitor instrumentation health

Instrumentation QA does not end when the release ships. A valid event can still become misleading when volume drops, identifiers disappear, property types drift, a new enum value arrives, or one application version stops emitting.

This guide turns a tracking plan into a small production monitoring system.

Define what healthy means#

Choose a short list of critical events. A practical starting set is:

• account or user created
• activation completed
• primary product action completed
• payment or subscription state changed
• important background job completed

For each event, record:

owner
source service
expected environments
required identifiers
required properties and types
known categorical values
expected daily or hourly volume range
maximum useful delivery delay
schema version
reconciliation source

Do not alert on every exploratory event. Monitor the contracts whose failure would change a decision or hide a customer problem.

Build a health dashboard#

Create a dashboard named Instrumentation health with four sections:

1. critical event volume
2. required-field completeness
3. schema and enum drift
4. delivery and freshness indicators

Put the owner and response link in the dashboard description. A health dashboard without an owner becomes another untrusted dashboard.

Detect event-volume drops#

Start with a daily count:

SELECT
  toStartOfDay(toDateTime(timestamp)) AS day,
  countIf(JSONExtractString(json, 'event') = 'signup_completed') AS signups,
  countIf(JSONExtractString(json, 'event') = 'activation_completed') AS activations,
  countIf(JSONExtractString(json, 'event') = 'subscription_started') AS subscriptions
FROM product_events
WHERE timestamp >= now() - INTERVAL 30 DAY
GROUP BY day
ORDER BY day

Compare like with like. A weekend drop may be expected; a release-hour drop isolated to one service or application version may not be.

For a critical single event, create a Single Line visualization in the collection visualizer and add an alert for an absolute or relative drop. Use a window large enough to avoid paging on normal minute-to-minute noise.

Volume alerts answer “did events arrive?” They do not prove the events contain the fields required for analysis.

Measure required-field completeness#

Use conditional counts for identifiers:

SELECT
  toStartOfDay(toDateTime(timestamp)) AS day,
  count() AS events,
  countIf(
    empty(JSONExtractString(json, 'account_id'))
  ) AS missing_account_id,
  countIf(
    empty(JSONExtractString(json, 'user_id'))
  ) AS missing_user_id,
  round(missing_account_id / events * 100, 2) AS missing_account_pct,
  round(missing_user_id / events * 100, 2) AS missing_user_pct
FROM product_events
WHERE
  JSONExtractString(json, 'event') = 'report_exported'
  AND timestamp >= now() - INTERVAL 30 DAY
GROUP BY day
ORDER BY day

An empty string and a missing key both appear empty through JSONExtractString. If empty is a legitimate value, introduce an explicit presence flag or validate before ingestion.

Track identifiers separately. user_id may be optional for account-level system events while account_id remains required.

Find type drift#

GraphJSON accepts flexible JSON, so the same property can arrive as more than one type. Use ClickHouse JSON type inspection:

SELECT
  JSONType(json, 'amount') AS amount_type,
  count() AS rows
FROM product_events
WHERE
  JSONExtractString(json, 'event') = 'checkout_completed'
  AND timestamp >= now() - INTERVAL 7 DAY
GROUP BY amount_type
ORDER BY rows DESC

For money, the expected result might be one integer type. A new String row often means one producer started sending "4900" or "$49.00".

Repeat this check for:

• numeric metrics
• booleans
• identifiers that must remain strings
• arrays or nested objects used in SQL

Fix the producer first. Query-time coercion can keep an investigation moving, but it should not silently become the permanent contract.

Detect unexpected categorical values#

List recently observed values:

SELECT
  JSONExtractString(json, 'plan') AS plan,
  count() AS rows,
  max(toDateTime(timestamp)) AS last_seen
FROM product_events
WHERE
  JSONExtractString(json, 'event') = 'subscription_started'
  AND timestamp >= now() - INTERVAL 30 DAY
GROUP BY plan
ORDER BY rows DESC

Compare the result with the tracking plan. Investigate:

• new legitimate values missing from documentation
• capitalization changes
• internal or test values in production
• empty strings
• deprecated values that should no longer arrive

When the value is legitimate, update the tracking plan and downstream queries in the same change.

Monitor schema-version adoption#

During a breaking instrumentation migration, count versions by application release:

SELECT
  JSONExtractInt(json, 'schema_version') AS schema_version,
  JSONExtractString(json, 'app_version') AS app_version,
  count() AS rows,
  min(toDateTime(timestamp)) AS first_seen,
  max(toDateTime(timestamp)) AS last_seen
FROM product_events
WHERE
  JSONExtractString(json, 'event') = 'checkout_completed'
  AND timestamp >= now() - INTERVAL 14 DAY
GROUP BY schema_version, app_version
ORDER BY schema_version, rows DESC

Do not remove compatibility logic merely because the new version appeared once. Define an adoption threshold and an expected retirement date for the old producer.

For mobile and desktop applications, old versions may continue emitting for months. Either support both shapes in the metric or enforce an application upgrade policy.

Measure delivery delay#

GraphJSON stores the occurrence timestamp supplied by the producer. To measure delivery delay precisely, add a second timestamp inside the event:

{
  "event": "report_generated",
  "event_id": "evt_01J...",
  "occurred_at": 1785081000,
  "emitted_at": 1785081012,
  "schema_version": 1
}

When a queue worker sends the event, add delivered_at to a separate delivery diagnostic event or record queue age in the worker:

{
  "event": "analytics_batch_delivered",
  "oldest_event_age_seconds": 42,
  "event_count": 50,
  "attempt": 2
}

Monitor the oldest event age, retry volume, rejected events, and dead-letter count. A healthy destination cannot compensate for an unhealthy producer queue.

Reconcile against a source of truth#

For business-critical facts, compare a settled interval:

completed orders in primary database
unique order_completed event_ids in GraphJSON
difference and known exclusions

Reconcile by stable ID, not only aggregate revenue. Equal totals can hide one missing event and one duplicate.

Do not make GraphJSON the authoritative ledger for billing, balances, entitlement, or order state. Send an analytical copy and reconcile it against the business system.

Triage a failed check#

Use this order:

1. Confirm the definition. Was the expected volume or enum list still correct?
2. Inspect Samples. Did the event arrive with a surprising shape?
3. Split by source. Compare environment, service, application version, and schema version.
4. Inspect delivery. Check response codes, retries, queue age, and dead letters.
5. Compare the source system. Determine whether the business action itself changed.
6. Contain bad data. Pause a replay or producer when continuing would expand the problem.
7. Repair and annotate. Record the affected interval and query treatment.

Never “fix” a health chart by weakening the expected contract without recording why.

Review cadence#

Pair this guide with Tracking plan and instrumentation QA, Reliable event delivery, and Environments and testing.