Build a durable raw archive and dual-write pipeline

GraphJSON is an analytical destination. If your recovery, audit, or migration requirements need an independently replayable copy, preserve one before or alongside delivery.

Do not implement this as two unrelated network calls in a user request:

business commit → call GraphJSON
                → call object storage

Either call can succeed alone. A retry can duplicate one destination, and a process crash can lose the other.

Use one durable fact and independent consumers:

business transaction
      ↓
transactional outbox or durable event log
      ├── GraphJSON consumer → normalized analytical event
      └── archive consumer   → immutable object-store segment

Decide whether you need an archive#

An independent raw archive is useful when you must:

• rebuild analytics after a mapping bug
• migrate to a new schema or destination
• recompute historical derived events
• retain source facts under a governed legal policy
• verify completeness independently of a vendor
• recover from accidental analytical collection changes

It also increases privacy surface, access-control work, storage cost, and deletion obligations. “Keep everything forever” is not a recovery strategy.

GraphJSON does not currently expose a whole-workspace raw archive export that should be treated as your sole disaster-recovery path. Design the source pipeline to meet your durability requirement.

Define the canonical envelope#

Archive the stable fact before destination-specific flattening:

{
  "event_id": "evt_01J...",
  "event": "report_published",
  "occurred_at": 1785081600,
  "observed_at": 1785081603,
  "source": "reporting-api",
  "environment": "production",
  "account_id": "acct_7",
  "schema_version": 2,
  "payload": {
    "report_id": "rpt_42",
    "format": "dashboard"
  }
}

Required envelope fields:

• globally stable event ID
• business occurrence time
• source and environment
• schema version
• tenant or subject key when allowed
• validated bounded payload

Add a record checksum in the archive manifest rather than allowing every producer to invent its own serialization checksum.

Do not archive credentials, session tokens, raw payment instruments, or unreviewed arbitrary request bodies.

Commit the fact once#

For application-owned business events, write an outbox row in the same database transaction as the business state:

BEGIN
  update report state to published
  insert canonical event into analytics_outbox
COMMIT

The request can then return without waiting for two analytical destinations. Workers claim outbox rows, retry with backoff, and checkpoint each destination.

If the source is already a durable stream, it can serve as the canonical log. The critical property is that consumers can resume from an acknowledged position without relying on process memory.

Keep destination checkpoints independent#

One logical event needs separate delivery state:

{
  "event_id": "evt_01J...",
  "graphjson": {"status":"delivered","attempts":1},
  "archive": {"status":"pending","attempts":3}
}

Do not mark the outbox complete merely because GraphJSON accepted the row while the archive is still missing. Do not block GraphJSON progress because one archive object is temporarily unavailable; each consumer needs its own bounded retry and dead-letter path.

Write immutable archive segments#

A practical object layout:

s3://analytics-archive/
  environment=production/
  collection=product_events/
  date=2026-07-26/
  hour=09/
  part-000184.ndjson.gz
  part-000184.manifest.json

Use event occurrence date for discoverability, but record write time in the manifest so late delivery is visible.

Do not let a client provide raw object keys. Normalize collection and partition values through a fixed registry.

Include a manifest#

Example:

{
  "format": "ndjson",
  "compression": "gzip",
  "schema": "canonical-event-v2",
  "record_count": 5000,
  "min_occurred_at": 1785081600,
  "max_occurred_at": 1785085199,
  "first_event_id": "evt_01J...",
  "last_event_id": "evt_01K...",
  "sha256": "8d4c..."
}

Write the data object to a temporary unique key, verify length and checksum, then publish the final object and manifest using the storage system’s atomic or conditional-write capabilities. Never overwrite a completed segment in place.

Choose NDJSON or Parquet deliberately#

Many teams keep a canonical compressed NDJSON archive and create governed Parquet derivatives. If Parquet is the only raw copy, test that older and newer schema versions can still be read together.

Small objects are expensive to list and process. Buffer by bounded size or short time, but cap the window so a crash does not hold excessive unflushed data. The canonical outbox remains the recovery source until the segment is confirmed.

Deliver normalized events to GraphJSON#

The GraphJSON consumer:

1. reads a bounded batch from the durable source
2. validates the canonical schema
3. maps it to the registered GraphJSON collection and shape
4. groups one collection per request
5. sends at most 50 rows through /api/bulk-log
6. preserves each original Unix-second timestamp
7. checkpoints only after a successful response
8. dead-letters permanent contract failures

Keep event_id, schema_version, and useful source provenance. The endpoint does not promise exactly-once delivery, so downstream exact metrics should deduplicate stable event IDs where necessary.

Reconcile destinations#

Counts alone are insufficient. Compare by partition:

GraphJSON event count may exceed distinct event count after a timeout and retry. Reconciliation should distinguish expected at-least-once duplicates from missing source facts.

Emit a compact archive_partition_reconciled operational event only after the independent reconciliation job finishes. Do not make the pipeline’s only health signal depend on the same worker it is measuring.

Replay without corrupting production#

Before a replay:

1. identify exact manifests and event-time interval
2. verify authorization and deletion holds
3. pin the mapping and schema versions
4. use a new collection or explicit migration marker
5. dry-run counts, sizes, and routing
6. send with conservative bounded concurrency
7. reconcile against the manifest
8. migrate readers only after acceptance

Example replay metadata:

{
  "migration_id": "replay_2026_07_plan_v3",
  "source_manifest": "part-000184.manifest.json",
  "source_event_id": "evt_01J..."
}

Do not replace the original event ID with a new random ID. Keep migration metadata separate from the identity used for deduplication.

Apply privacy and security controls twice#

The archive and GraphJSON are separate data stores. Both need:

• access review and least privilege
• encryption in transit and at rest
• auditability
• regional and contractual controls
• retention enforcement
• legal hold behavior
• subject export and deletion procedures

Object versioning or immutability can conflict with deletion obligations. Choose retention-lock settings with legal and security owners, not merely as a technical default.

Use separate credentials and failure domains where practical. An archive in the same account, region, permission role, and deletion automation as its source may not protect against the incident you care about.

Test recovery#

At least periodically:

1. select a small completed manifest
2. restore it to an isolated workspace and collection
3. run contract and reconciliation checks
4. compare a known metric with the original
5. measure time to restore
6. document missing access, tooling, or instructions

A backup that has never been restored is an assumption. Record recovery point and recovery time expectations, then test against them.

Production checklist#

• A canonical event is committed once to a durable source.
• GraphJSON and archive consumers have independent checkpoints.
• Stable event IDs and original event times survive every destination.
• Archive objects are immutable, checksummed, and manifested.
• Partition layout and object size are bounded.
• Schema evolution is readable across retained history.
• Reconciliation checks IDs, time ranges, values, and versions.
• Replay uses an isolated destination and explicit migration ID.
• Archive and GraphJSON privacy lifecycles are both implemented.
• A real restore is tested on a documented cadence.

Continue with reliable event delivery, high-volume ingestion and backfills, and export, deletion, and portability.