LLM product and cost analytics

An LLM call is both a product event and an operational event. It represents a customer using a feature, a provider request consuming tokens and money, and a probabilistic result whose latency and quality may change by model or prompt version.

GraphJSON can analyze those facts as structured JSON. It is not a replacement for full distributed tracing, prompt inspection, or an evaluation platform.

Decide what the dashboard must answer#

A useful first version answers:

• Which accounts and users adopt each AI feature?
• Which models and providers handle the workload?
• What do input, output, cached, and reasoning tokens cost?
• Where are latency and failure rates increasing?
• Did a prompt, model, or release change quality?
• Which usage is internal, test, or customer-facing?

Avoid logging every possible provider field before these questions have an owner.

Emit one terminal generation event#

Log after the provider call finishes:

{
  "event": "llm_generation_completed",
  "request_id": "llmreq_01J...",
  "account_id": "acct_7",
  "user_id": "usr_42",
  "feature": "support_reply_draft",
  "provider": "example_provider",
  "model": "model-v3",
  "prompt_version": "support-draft-12",
  "release": "web-2026.07.26.1",
  "environment": "production",
  "success": true,
  "input_tokens": 1840,
  "cached_input_tokens": 1200,
  "output_tokens": 318,
  "reasoning_tokens": 0,
  "duration_ms": 1460,
  "time_to_first_token_ms": 390,
  "cost_microusd": 1720,
  "cost_source": "provider",
  "schema_version": 1
}

Use integer microdollars when sub-cent precision matters:

1 USD = 1,000,000 microusd

Do not combine inclusive and exclusive token counters. Decide whether input_tokens includes cached tokens and document the contract. The example treats input_tokens as the uncached input bucket and cached_input_tokens as a separate, non-overlapping bucket.

Prefer provider-reported usage and cost. If you estimate cost, set cost_source to estimated and preserve the pricing-table version used by your application. Model prices change; silently applying today’s price to historical requests rewrites the past.

Emit failures with the same contract#

Keep the event name stable or use a separate failure event according to the alerting workflow:

{
  "event": "llm_generation_failed",
  "request_id": "llmreq_01J...",
  "account_id": "acct_7",
  "feature": "support_reply_draft",
  "provider": "example_provider",
  "model": "model-v3",
  "prompt_version": "support-draft-12",
  "release": "web-2026.07.26.1",
  "environment": "production",
  "success": false,
  "error_category": "rate_limited",
  "status_code": 429,
  "duration_ms": 812,
  "attempt": 2,
  "schema_version": 1
}

Store a controlled error_category, not the complete provider response. Provider errors can echo prompts, user content, URLs, or credentials.

Recommended categories:

rate_limited
timeout
provider_5xx
authentication
invalid_request
content_policy
cancelled
unknown

Keep prompts and outputs out by default#

Product analytics rarely needs raw prompt or response text.

Do not send:

• system prompts
• user messages
• model responses
• retrieved documents
• tool arguments or outputs
• complete provider request or response objects
• API keys or authorization headers

Use controlled metadata:

{
  "prompt_version": "support-draft-12",
  "feature": "support_reply_draft",
  "language": "en",
  "input_character_bucket": "1000-4999",
  "retrieval_used": true,
  "tool_count": 2
}

If raw content is required for debugging or evaluation, store it in a purpose-built restricted system with its own access, retention, and redaction controls. Keep an opaque request_id in both systems.

Track feedback separately#

Feedback often arrives after generation:

{
  "event": "llm_feedback_recorded",
  "request_id": "llmreq_01J...",
  "account_id": "acct_7",
  "feature": "support_reply_draft",
  "feedback": "accepted",
  "edited_before_use": true,
  "schema_version": 1
}

Controlled feedback values might be:

accepted
rejected
thumbs_up
thumbs_down
regenerated

Do not infer quality from one proxy alone. A response can be accepted because the user is rushed, regenerated for stylistic preference, or never rated.

Query adoption#

SELECT
  toStartOfWeek(toDateTime(timestamp)) AS week,
  JSONExtractString(json, 'feature') AS feature,
  uniqExact(JSONExtractString(json, 'account_id')) AS active_accounts
FROM llm_events
WHERE
  JSONExtractString(json, 'event') = 'llm_generation_completed'
  AND JSONExtractBool(json, 'success') = 1
  AND JSONExtractString(json, 'environment') = 'production'
  AND timestamp >= now() - INTERVAL 90 DAY
GROUP BY week, feature
ORDER BY week, feature

Visualize as Multi Line with:

timestamp: week
value: active_accounts
split: feature

Query cost by model#

SELECT
  JSONExtractString(json, 'provider') AS provider,
  JSONExtractString(json, 'model') AS model,
  sum(JSONExtractInt(json, 'cost_microusd')) / 1000000 AS cost_usd,
  count() AS generations
FROM llm_events
WHERE
  JSONExtractString(json, 'event') = 'llm_generation_completed'
  AND JSONExtractString(json, 'environment') = 'production'
  AND timestamp >= now() - INTERVAL 30 DAY
GROUP BY provider, model
ORDER BY cost_usd DESC

Keep provider and model as separate fields. The same model label can exist behind multiple providers or commercial arrangements.

Query cost per successful generation#

SELECT
  JSONExtractString(json, 'feature') AS feature,
  round(
    sum(JSONExtractInt(json, 'cost_microusd')) /
    countIf(JSONExtractBool(json, 'success') = 1),
    2
  ) AS microusd_per_success
FROM llm_events
WHERE
  JSONExtractString(json, 'event') = 'llm_generation_completed'
  AND timestamp >= now() - INTERVAL 30 DAY
GROUP BY feature
ORDER BY microusd_per_success DESC

Guard against zero successful rows when adapting this query for sparse features.

Query latency and failures#

SELECT
  JSONExtractString(json, 'feature') AS feature,
  JSONExtractString(json, 'model') AS model,
  quantile(0.5)(JSONExtractFloat(json, 'duration_ms')) AS p50_ms,
  quantile(0.95)(JSONExtractFloat(json, 'duration_ms')) AS p95_ms,
  count() AS generations
FROM llm_events
WHERE
  JSONExtractString(json, 'event') = 'llm_generation_completed'
  AND timestamp >= now() - INTERVAL 7 DAY
GROUP BY feature, model
ORDER BY p95_ms DESC

Failure rate:

SELECT
  toStartOfHour(toDateTime(timestamp)) AS hour,
  countIf(JSONExtractString(json, 'event') = 'llm_generation_failed') AS failures,
  count() AS attempts,
  round(failures / attempts * 100, 2) AS failure_pct
FROM llm_events
WHERE
  JSONExtractString(json, 'event') IN (
    'llm_generation_completed',
    'llm_generation_failed'
  )
  AND timestamp >= now() - INTERVAL 7 DAY
GROUP BY hour
ORDER BY hour

Build the dashboard#

Recommended tiles:

1. weekly active accounts by AI feature
2. generations by model
3. cost by feature
4. cost per successful generation
5. P50 and P95 latency
6. failures by category
7. accepted or positive-feedback rate
8. model and prompt-version comparison

Every cost tile should state whether the source is provider-reported or estimated.

Create actionable alerts#

Use collection Single Line visualizations for:

• count of llm_generation_failed
• P95 duration_ms for completed generations
• total cost_microusd
• count of content_policy failures

Set thresholds per feature or model when the baseline differs substantially. Put the response in the alert description:

Hourly generation failures exceed 50.
Check provider status, rate limits, the latest release, and queue retries.

GraphJSON alerts are not an automatic provider failover mechanism. Your application owns fallback, retry, and circuit-breaker behavior.

Compare releases, prompts, and models#

Include immutable dimensions:

• release
• prompt_version
• provider
• model

Compare adoption, cost, latency, failures, and feedback together. A cheaper model that increases rejection and regeneration can cost more per accepted outcome.

Do not interpret a before/after chart as a controlled experiment when traffic, feature population, or seasonality also changed.

Production checklist#

• Raw prompts, responses, and tool payloads stay out of analytics.
• Token buckets are non-overlapping and documented.
• Cost uses integer units and a declared source.
• Provider, model, prompt version, feature, release, and environment are present.
• Failures use controlled categories.
• Feedback is linked by opaque request ID.
• Internal and test usage is excluded from customer cost metrics.
• Authoritative provider invoices are reconciled separately.
• Alerts have owners and response steps.
• Retention matches the sensitivity and useful lifetime of the data.