Migrate historical analytics data

A reliable analytics migration has two separate goals:

1. new events arrive correctly after the cutover
2. the historical window needed for trends, cohorts, and customer context remains usable

Do not combine those into one irreversible switch. Build and validate the new production path first, import history through a resumable pipeline, compare both systems, and retire the old pipeline only after the metrics agree within a documented tolerance.

Choose the history you actually need#

Importing every historical row is not automatically the safest choice. Old tracking plans often contain duplicate names, sensitive properties, obsolete schemas, and events nobody uses.

Inventory the source before exporting:

For many teams, 12–18 months of a curated event set is more useful than an unreviewed lifetime export.

Design the target contract first#

Map source fields into the same contract your new GraphJSON instrumentation uses. A portable target event might be:

{
  "event": "subscription_started",
  "event_id": "src_01J...",
  "user_id": "usr_42",
  "account_id": "acct_7",
  "plan": "pro",
  "schema_version": 2,
  "migration_source": "legacy_vendor"
}

Keep the original occurrence time in the request-level timestamp, expressed as a whole Unix second. Do not replace it with export or import time.

Preserve a source-generated unique event ID when one exists. Put vendor-specific bookkeeping behind an explicit prefix such as source_insert_id; do not let reserved vendor names become your permanent schema by accident.

Use a staged migration#

The safest flow separates extraction from ingestion:

source export
    ↓
immutable staged files
    ↓
deterministic transform
    ↓
validation and quarantine
    ↓
GraphJSON bulk import
    ↓
count and metric reconciliation

Store source exports in encrypted object storage with restricted access and a defined deletion date. Record a checksum, source date range, export time, and row count for every file. A durable staging layer lets you fix transformation or delivery code without repeatedly querying the source.

Transform deterministically: the same source record should always produce the same destination JSON and event_id. Invalid rows belong in a quarantine file with a machine-readable reason, not in console output or silent discard.

Import in bounded batches#

GraphJSON’s bulk endpoint accepts up to 50 events for one collection:

async function sendBatch(events) {
  const response = await fetch("https://api.graphjson.com/api/bulk-log", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      api_key: process.env.GRAPHJSON_API_KEY,
      collection: "product_events",
      timestamps: events.map((item) => item.timestamp),
      jsons: events.map((item) => JSON.stringify(item.body)),
    }),
  });

  const responseBody = await response.text();
  if (!response.ok) {
    throw new Error(`GraphJSON ${response.status}: ${responseBody}`);
  }
}

Build the importer as a restartable worker:

• checkpoint after each acknowledged batch
• keep a batch until GraphJSON returns 2xx
• retry network failures, 429, and 5xx with exponential backoff and jitter
• do not retry an unchanged 400
• split failed batches to isolate invalid events
• cap concurrency and respect the 40-request-per-second workspace guardrail
• retain a dead-letter file for manual review

The endpoint does not provide an idempotency key or exactly-once guarantee. A timeout can leave the sender uncertain about the outcome. Preserve event_id and use a deduplicating query when retries may have created duplicate rows.

Pilot before the full import#

Start with one representative day that includes:

• ordinary activity
• users and accounts with multiple events
• boundary timestamps near midnight
• sparse optional fields
• the largest expected payload shape
• source-specific reserved fields

Inspect the imported rows in Samples, run the target queries, and verify that the date appears on the intended calendar day and time zone. Fix the transform before increasing the window.

Dual-write new activity#

Once the GraphJSON event contract is stable, send new production events to both systems for a defined comparison period. Keep the send paths independently observable so a failure in the legacy destination does not hide the GraphJSON result, or vice versa.

Avoid deriving GraphJSON events by polling the old analytics vendor unless that is your permanent architecture. Direct server-side instrumentation gives you clearer semantics and fewer delivery dependencies.

Reconcile from raw facts#

Compare source and GraphJSON by event name and UTC day:

SELECT
  toDate(toDateTime(timestamp, 'UTC')) AS day,
  JSONExtractString(json, 'event') AS event,
  count() AS rows,
  uniqExact(JSONExtractString(json, 'event_id')) AS unique_events
FROM product_events
WHERE JSONExtractString(json, 'migration_source') != ''
GROUP BY day, event
ORDER BY day, event

Reconciliation should include:

• row and unique-event counts by day
• minimum and maximum event time
• missing user_id and account_id rates
• numeric totals such as revenue or duration
• duplicate event_id counts
• invalid and quarantined rows
• a sample of complete user or account journeys

Vendor dashboards often apply identity merging, timezone rules, bot filters, late-arrival behavior, or proprietary attribution. Compare the raw export to raw GraphJSON events first. Only then compare derived metrics with equivalent definitions.

Cut over and decommission#

Cut over only when:

• direct GraphJSON ingestion has run through a representative production period
• required dashboards and alerts have replacements
• the historical import is complete or its exclusions are documented
• reconciliation is signed off by the metric owners
• rollback and support contacts are known

After cutover, keep the old destination read-only for the contractual or operational period you need. Remove legacy SDKs and secrets deliberately, stop export jobs, and delete staged files on the retention date.

Provider runbooks#

• Migrate from Amplitude
• Migrate from Mixpanel
• Migrate from PostHog
• Migrate from Google Analytics 4

For a migration from another system, use this workflow and write a source mapping for its event name, occurrence time, identifiers, properties, and unique-event key before importing.