Collection architecture at scale

A collection is both a query table and an operational boundary. A good collection groups events that belong together without forcing unrelated data to share access, retention, or ingestion behavior.

Start with a small number of domain collections. Split only when a concrete boundary requires it.

Use six decision axes#

Before creating a collection, answer:

Events that differ materially on two or more axes usually deserve separate collections or workspaces.

Start with domain collections#

A practical starting layout:

product_events
billing_events
integration_events
application_logs

Possible additions:

account_state_events
usage_events
public_metrics

product_events can hold page-independent product actions such as report_created, dashboard_shared, and invite_accepted. It should not become a dumping ground for request logs, raw webhook payloads, or invoice snapshots merely because all of them have an account_id.

Avoid collection-per-event design#

This layout is usually too granular:

signup_completed
report_created
report_published
dashboard_shared

It increases producer configuration, query unions, migrations, retention settings, and inventory work. Related product actions typically belong in one product_events collection and use an event property as a low-cardinality dimension.

Create a new collection when it changes the lifecycle or trust boundary, not just because it adds a new event name.

Avoid one collection for everything#

An unbounded events collection creates the opposite problem:

• operational logs drown out product actions
• a broad reader can see data they do not need
• short-lived debug fields inherit long-lived retention
• producer mistakes have a larger blast radius
• common queries scan unrelated rows
• schema and owner become unclear

all_events is already available as a workspace-wide virtual table for occasional cross-collection analysis. You do not need to collapse every domain into one physical collection to query them together.

Name collections predictably#

Prefer stable lowercase snake_case names:

product_events
billing_events
application_logs

Names should describe the domain, not:

• an individual customer
• a person or email address
• a temporary project codename
• a month or day
• a deployment ID
• an event type that will soon have siblings

Keep the deployment environment out of a production collection name when separate workspaces already provide the environment boundary. If one non-production workspace intentionally holds several ephemeral environments, use bounded prefixes such as ci_product_events; do not create arbitrary user-controlled collection names.

Separate environments strongly#

Recommended:

production workspace
staging workspace
CI workspace

Workspace isolation prevents a missing environment = 'prod' filter from mixing test traffic into a real metric. It also gives credentials, membership, retention, and destructive lifecycle actions independent blast radii.

An environment property remains useful for diagnostics, but it is not a substitute for a hard boundary.

Separate by sensitivity, not just topic#

Product actions and application logs may describe the same workflow but need different readers. Logs can contain request paths, error classes, and internal diagnostics. Billing facts can contain amounts and invoice references.

Use the narrowest collection that lets an operator do their job. Do not copy secret values, credentials, raw payment instruments, message bodies, or unreviewed arbitrary payloads into any collection.

GraphJSON workspace roles and API keys are important, but a collection should not be treated as fine-grained row-level authorization. Use separate workspaces when organizations need a genuine administrative or data-boundary separation.

Do not create a collection per tenant by default#

A tenant-per-collection layout creates unbounded tables and configuration:

product_events_acct_1
product_events_acct_2
product_events_acct_3

For shared analytics, use a stable account_id on every tenant-scoped fact and enforce access in the application layer. For strict regulatory, contractual, or regional isolation, use a deliberate workspace or system boundary—not a dynamically chosen browser collection.

Never accept a collection name from an untrusted client.

Choose raw versus normalized collections#

An integration often has two useful representations:

provider payload → durable raw archive
                 → validated normalized GraphJSON event

The normalized event should expose only fields needed for analysis:

{
  "event": "invoice_payment_failed",
  "event_id": "evt_provider_91",
  "account_id": "acct_7",
  "invoice_id": "inv_42",
  "amount_due_minor": 4900,
  "currency": "usd",
  "attempt_number": 2,
  "schema_version": 1
}

Do not put an entire third-party webhook into GraphJSON and call it future proofing. Raw payloads have a broader privacy surface, less stable schema, and different archive requirements. Keep them in a governed durable source when you need replay.

Align retention and volume#

High-volume attempt logs may need days or weeks; compact terminal product facts may support year-over-year analysis. If two event groups need different retention or sampling, separating collections can make that policy legible.

Estimate:

daily events
× average serialized bytes
× retained days
× expected growth factor

Then include query scan cost and backfill behavior. A collection layout is not finished until volume and lifecycle owners agree on it.

For noisy streams, store compact terminal outcomes longer and detailed attempts for less time. Never sample facts used for billing, quota enforcement, security, or exact reconciliation.

Plan cross-collection analysis#

Join or union domains through stable keys:

• account_id
• user_id
• event_id
• request_id
• subscription_id
• integration_delivery_id

Normalize key types and time semantics at ingestion. A join between an integer ID in one collection and a formatted string in another is avoidable friction.

Pre-aggregate each side before a large join:

WITH
  product_by_account AS (
    SELECT
      JSONExtractString(json, 'account_id') AS account_id,
      count() AS product_actions
    FROM product_events
    WHERE timestamp >= now() - INTERVAL 30 DAY
    GROUP BY account_id
  ),
  billing_by_account AS (
    SELECT
      JSONExtractString(json, 'account_id') AS account_id,
      sum(JSONExtractInt(json, 'amount_minor')) AS paid_minor
    FROM billing_events
    WHERE
      timestamp >= now() - INTERVAL 30 DAY
      AND JSONExtractString(json, 'event') = 'invoice_paid'
    GROUP BY account_id
  )
SELECT *
FROM product_by_account
LEFT JOIN billing_by_account USING account_id

This prevents a many-to-many row explosion before aggregation.

Maintain a collection registry#

Keep a source-controlled record:

name: application_logs
owner: platform
purpose: terminal server request and job outcomes
allowed_data: operational metadata; pseudonymous account_id
prohibited_data: credentials; message bodies; payment instruments
retention_class: short_operational
producers:
  - api
  - background_workers
key_fields:
  - event
  - event_id
  - request_id
  - account_id
schema_versions:
  - 1

Also record the workspace, API-key owner, expected daily range, instrumentation health query, critical consumers, and retirement state.

Split a collection safely#

When events must become product_events and application_logs:

1. define routing rules with no ambiguous event names
2. create destination contracts and collections
3. publish or replay the same stable event IDs
4. compare source and destination by day, event, count, distinct ID, and key sums
5. migrate saved queries, dashboards, alerts, embeds, and exports
6. stop new writes to the old path
7. preserve the old history for the agreed rollback or compliance window
8. record the historical cutover in metric definitions

Avoid “temporary” dual remote writes inside the request path. A durable outbox or queue should own fan-out and retry.

Merge collections carefully#

Merge only when the source domains now share owner, sensitivity, retention, environment, volume, and analytical use. Resolve conflicting event names and field types first.

Keep source_collection or a migration provenance field during a replay when operators need to reconcile. Do not retain migration metadata forever if it has no durable analytical purpose.

New collection checklist#

• The domain and owner are unambiguous.
• Sensitivity and prohibited fields are documented.
• Workspace and environment boundary are explicit.
• Retention and expected daily volume are approved.
• Event names and key types follow the tracking plan.
• Collection name is stable and bounded.
• Tenant identity is a property, not an untrusted collection selector.
• Cross-collection keys and time semantics are compatible.
• Delivery, dead-letter, and instrumentation-health monitoring are defined.
• Critical consumers and retirement procedure are in the registry.

Continue with events and collections, capacity, retention, and cost, and high-volume ingestion and backfills.