LLM Analytics: Measure Cost, Latency, Adoption, and Quality

An AI feature creates three kinds of questions at once.

Product wants to know whether customers use it. Engineering wants to know whether the provider is slow or failing. Finance wants to know why the model bill doubled.

The provider dashboard usually answers only part of the third question. It can show tokens by model, but not which customer, feature, prompt version, or release created them. Application logs may contain the missing context, but raw logs are awkward to group into weekly adoption, cost per successful outcome, or P95 latency by feature.

The useful middle ground is one structured event per model call.

Log the outcome, not the entire conversation#

After a generation finishes, emit:

{
  "event": "llm_generation_completed",
  "request_id": "llmreq_01J...",
  "account_id": "acct_7",
  "feature": "support_reply_draft",
  "provider": "example_provider",
  "model": "model-v3",
  "prompt_version": "support-draft-12",
  "release": "web-2026.07.26.1",
  "success": true,
  "input_tokens": 1840,
  "cached_input_tokens": 1200,
  "output_tokens": 318,
  "duration_ms": 1460,
  "cost_microusd": 1720,
  "cost_source": "provider"
}

That is enough to answer:

• active accounts by AI feature
• cost by customer, feature, model, and release
• cost per successful generation
• token mix and cache use
• P50 and P95 latency
• failure rate by provider
• model adoption after a rollout

It is deliberately not a trace. The event does not contain the system prompt, user message, output, retrieved documents, or tool arguments.

Raw prompts are usually the wrong analytics payload#

Model input and output can contain customer conversations, source code, medical details, support tickets, credentials, or anything else a user pasted into the product.

Putting all of that into a broad analytics collection creates an access and retention problem in exchange for fields that most charts never use.

Use metadata instead:

{
  "feature": "support_reply_draft",
  "prompt_version": "support-draft-12",
  "language": "en",
  "retrieval_used": true,
  "tool_count": 2,
  "input_character_bucket": "1000-4999"
}

If your team needs prompt inspection or evaluation, use a purpose-built restricted system and connect it to analytics with an opaque request ID.

Token definitions need a contract#

Providers report token usage differently. Some totals include cached or reasoning tokens; others provide separate buckets. If you store an inclusive input count and a separate cached count, a dashboard that sums both will double-count.

Pick non-overlapping fields and document them:

input_tokens: uncached input only
cached_input_tokens: cache reads only
output_tokens: visible output only
reasoning_tokens: billed reasoning only

Prefer provider-reported cost. When you estimate it, store the price-table version and label the source estimated. Repricing old events using today’s model price silently changes historical cost.

For sub-cent values, integer microdollars avoid floating-point ambiguity:

1 USD = 1,000,000 microusd

Cost without outcome is incomplete#

Cheapest per call is not necessarily cheapest per useful result.

A smaller model may cut token cost and increase:

• retries
• user rejection
• regeneration
• manual editing
• support escalations

Track feedback as a later event linked by request ID:

{
  "event": "llm_feedback_recorded",
  "request_id": "llmreq_01J...",
  "feature": "support_reply_draft",
  "feedback": "accepted",
  "edited_before_use": true
}

Then compare cost, latency, failures, and accepted outcomes by model and prompt version. One proxy is never the full definition of quality, but it is much more useful than optimizing token price alone.

The first dashboard#

Start with eight tiles:

1. weekly active accounts by feature
2. generations by model
3. cost by feature
4. cost per successful generation
5. P50 and P95 latency
6. failures by controlled category
7. positive or accepted feedback rate
8. results by release and prompt version

Keep provider invoices authoritative. The analytics dashboard helps explain and allocate the bill; it should not silently become the bill.

The complete event schema, SQL, alert design, and privacy checklist are in the LLM product and cost analytics recipe.