Claude Dev Environment

Historical proposal. The Reaktor Shadow textbook pages and Operations & Reliability supersede this draft. In particular, moving-base publication differs from pinned replay, store materialization is explicit, deterministic fidelity is qualified, production read-through is denied and pinned cases obey deletion policy. The original prose below is preserved for lineage.

Dev runs against a continuously-refreshed shadow of production: the last 7 days of Supabase rows, Memgraph nodes, R2 objects, worker payloads, and JVM API traces — plus a hand-curated set of pinned crashes and anomalies that never expire. All testing, load testing, and exploratory debugging happens here. Built as a graph visitor (per Reaktor Spec §28) extended end-to-end across the actual data surfaces in BestBuds.

• Builds on — reaktor-shadow

• First consumer BestBuds dev cluster

• Window rolling 7-day + permanent pinned fixtures

• Drivers Keploy · k6 · Maestro

• Status Design

• Stateful surfaces — 7 (Postgres · Memgraph · R2 × 2 · Workers AI · Sheets · JVM)

• Cloudflare workers in scope — 10

• Storage budget — ~50 GB at peak

• Refresh cadence — hourly capture · nightly trim

• PII handling — scrub at capture (not query)

• Side-effect safety — tenant prefix + outbox interceptor

1. Executive summary

"Dev" today means an empty database and a stubbed worker — production bugs lived in a state of the world the staging environment never produced. The shadow dev environment fixes this by making production state the default state of dev: the same Supabase tables, the same Memgraph subgraphs, the same R2 objects, the same chat-message shapes, all updated continuously and trimmed to a rolling 7-day window. Any captured trace can be replayed against your branch — like Keploy, but covering all six Reaktor transports (HTTP · LOCAL · PEER · PUBSUB · QUEUE · WORKFLOW), every database, and the full causal chain of downstream calls a request triggers.

The architecture has three loops:

• Capture loop — a ShadowCaptureVisitor attaches to every ServiceNode, RepositoryNode, and WorkerBase in production. Every request/response pair, every DB write, every external call is encoded as a FlexBuffer envelope tagged with a capture.id and stored in R2 partitioned by hour. Logical replication mirrors Postgres into a shadow database. R2 reads through to production for cache misses.
• Refresh loop — nightly Worker trims captures older than 7 days, except for those marked pinned=true by either a human or the anomaly detector. Postgres shadow vacuums tombstoned rows; Memgraph snapshots roll forward.
• Replay loop — devs run shadow replay <capture-id> (CLI, also a Workbench button); the engine reconstructs the request context, deflects external API calls to deterministic stubs, executes against the branch, and diffs the response + causal chain against the captured baseline. Used in CI, load tests, and interactive debugging.

2. Current state

2.1 What dev/test infrastructure exists today

2.2 The honest gap

Today a dev with a branch has three options, all bad: (a) run nothing, write code blind; (b) wire their laptop to staging and hope they don't fat-finger a destructive query; (c) spin up an empty local Postgres + Memgraph + Workers and hand-craft fixtures that never resemble production. Option (c) is what most teams settle for, and it's why every "this worked in staging" bug exists.

3. Why a moving shadow

Bugs live in production shapes

The user with 4,200 chat messages, the group with 87 members, the friend graph with a 6-hop cycle — these don't appear in fixtures. Reproducing requires the actual data; recent actual data, because schemas and content drift week to week.

Load tests are useless without realistic mix

k6 scripts that hit /api/feed at uniform 100 rps don't predict the production failure mode where one user with 12,000 friends makes a single read that joins 12,000 rows. Real traffic patterns are the input; synthetic load is the multiplier.

Edge cases need permanence

A crash from January that took 6 hours to root-cause must be reproducible in October — not "if you can find the row in the backup." Pinned fixtures keep the weird ones forever.

Side effects need a sandbox

Devs experimenting can't be allowed to send real FCM notifications, write to the real Manna Google Sheet, or trigger Workers AI charges. Shadow stack means side effects are intercepted at the visitor.

Privacy by construction

PII scrubbed once at capture time beats scrubbing on every query. Shadow data is already safe to look at; devs don't need to remember to anonymise.

Replay closes the regression loop

Every captured request is a candidate test case. CI replays the last week of distinct traffic against the PR branch. If the response shape or P95 latency drifts, the PR is flagged — same model as Keploy, but covering all transports.

4. Prior art

5. Principles

1 · Capture is a visitor, not a proxy

Per spec §28.1: ShadowCaptureVisitor attaches at the graph layer. Every transport, every node, one mechanism. No sidecar, no service mesh, no eBPF.

2 · The shadow IS dev

There is no separate "dev environment". The shadow stack is dev. Devs run their branch against shadow workers, shadow Postgres, shadow Memgraph, shadow R2.

3 · Moving by default, pinned by exception

The default is a 7-day rolling window — every captured trace older than that is trimmed. Exceptions (crashes, anomalies, hand-curated edge cases) are pinned and never expire.

4 · Side effects fail loudly in shadow

Every external call (FCM, APNs, Workers AI, Google Sheets) hits a stub by default. Bypassing requires explicit --allow-real-effects + a confirmation prompt.

5 · PII scrubbed at capture

Names, emails, phone numbers, message bodies, location coordinates — replaced with deterministic fakes (same input → same fake) before anything hits R2. The shadow store contains no real PII.

6 · Replay must be deterministic

Time, RNG, IDs, external calls — all read from the captured envelope, not the system. A replay that gives a different answer means the code changed, not the environment.

7 · One refresh job per surface

Each stateful surface has its own refresh worker — Postgres has a logical-replication subscriber, Memgraph has a nightly snapshot importer, R2 has an event-driven mirror. Failures are isolated; one stuck refresh doesn't poison the rest.

8 · Tenant prefix everywhere

Shadow tenants live as shadow_$tenant — IDs prefixed, indexes prefixed, R2 keys prefixed. Prod queries with the wrong prefix hit nothing; shadow queries can't accidentally hit prod.

6. Stateful surfaces to shadow

From a fresh audit of bestbuds/targets/*Server, cloud/, and the JVM reaktorServer:

7. Side-effect classification

Replay correctness hinges on knowing which calls are safe to re-execute (idempotent reads, deterministic computations) vs which would produce duplicate real-world effects. Three categories, each with a default behaviour:

The visitor classifies on capture using two signals: HttpMethod for HTTP (GET/HEAD/OPTIONS → idempotent), and explicit @SideEffect(idempotent = true|false) annotations on Reaktor service handlers. Handlers without an annotation default to non-idempotent (safe default — better to over-stub than over-fire).

8. Stack diagram

┌────────────────────────── PRODUCTION ────────────────────────────────┐
│ user request                                                         │
│     │                                                                │
│     ▼                                                                │
│ ┌────────────────────────────────────────────────────────────────┐   │
│ │ Cloudflare Worker / JVM Service                               │   │
│ │ ┌────────────────────────────────────────────────────────────┐ │   │
│ │ │ ServiceNode                                                │ │   │
│ │ │ ├─ ShadowCaptureVisitor.intercept(req)                     │ │   │
│ │ │ │  ↓ writes envelope → R2 captures bucket                  │ │   │
│ │ │ ├─ RepositoryNode → Postgres (logical replication)          │─┼──┐
│ │ │ ├─ GraphNode → Memgraph (CDC + nightly dump)                │ │  │
│ │ │ └─ ActorNode → bus publish (outbox)                         │ │  │
│ │ └────────────────────────────────────────────────────────────┘ │   │
│ └────────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────│────────────────────────────────────┘
                                  │ replication / capture
                                  ▼
┌────────────────────────── SHADOW STACK ──────────────────────────────────┐
│ ┌──────────────────────┐ ┌──────────────────────┐ ┌──────────────────┐ │
│ │ shadow Postgres      │ │ shadow Memgraph      │ │ shadow R2        │ │
│ │ logical replica      │ │ snapshot + delta     │ │ read-through     │ │
│ │ 7d window + pinned   │ │ rebuilt nightly      │ │ + write capture  │ │
│ └──────────────────────┘ └──────────────────────┘ └──────────────────┘ │
│ ┌──────────────────────┐ ┌──────────────────────┐ ┌──────────────────┐ │
│ │ captures bucket      │ │ stubs registry       │ │ pin index        │ │
│ │ FlexBuffer envelopes │ │ AI / Sheets / FCM    │ │ DB-backed list   │ │
│ │ partitioned by hour  │ │ replay responses     │ │ of permanent IDs │ │
│ └──────────────────────┘ └──────────────────────┘ └──────────────────┘ │
│ ┌──────────────────────────────────────────────────────────────────────┐ │
│ │ Shadow workers (mirror of prod fleet)                                │ │
│ │ analyticsServer · chatServer · stickerServer · configServer ·        │ │
│ │ mediaServer · notificationServer · socialServer · messagingServer ·  │ │
│ │ botServer — all with shadow bindings                                 │ │
│ └──────────────────────────────────────────────────────────────────────┘ │
│                     ▲                                                    │
│                     │ dev branch / CI replay                             │
│ ┌───────────────────┴──────────────────────────────────────────────────┐ │
│ │ ShadowReplay engine · CLI · CI job · Workbench panel                 │ │
│ └──────────────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────┘
          ▲
          │ nightly trim · anomaly auto-pin
┌───────────────────────────┐
│ ShadowRefresh Worker      │
│ cron: 0 4 * * * UTC       │
└───────────────────────────┘

9. Capture path

9.1 The visitor

class ShadowCaptureVisitor(
  private val sink: CaptureSink,
  private val scrubber: PiiScrubber,
  private val sampler: CaptureSampler,
) : Visitor() {

  override fun visitServiceNode(node: ServiceNode): ExitScope {
    node.intercept(object : ServiceInterceptor {
      override suspend fun intercept(req: Request, chain: ServiceChain): Response {
        if (!sampler.shouldCapture(req)) return chain.proceed(req)

        val captureId = Uuid.v7().toString()
        val downstream = mutableListOf<DownstreamCall>()

        // Push a CaptureContext into the coroutine context so that child
        // service / DB / actor calls all append to `downstream`.
        return withCaptureContext(CaptureContext(captureId, downstream)) {
          val startedAt = Clock.System.now()
          val response = try { chain.proceed(req) }
                         catch (t: Throwable) { sink.captureError(captureId, t); throw t }
          val finishedAt = Clock.System.now()

          sink.write(CaptureEnvelope(
            id           = captureId,
            tenantId     = req.tenantId,
            transport    = req.transport,
            request      = scrubber.scrub(req),
            response     = scrubber.scrub(response),
            downstream   = downstream.toList(),
            startedAt    = startedAt,
            durationMs   = (finishedAt - startedAt).inWholeMilliseconds,
            handlerNode  = node.id,
          ))

          response
        }
      }
    })
    return NoOpExit
  }

  override fun visitRepositoryNode(node: RepositoryNode): ExitScope { /* append to downstream */ }
  override fun visitActorNode(node: ActorNode):           ExitScope { /* append to downstream */ }
}

9.2 What an envelope contains

{
  "id":           "01HXKQF8M7...",        // UUID v7 (time-sortable)
  "tenantId":     "shadow_bestbuds",
  "transport":    "HTTP",                   // HTTP | LOCAL | PEER | PUBSUB | QUEUE | WORKFLOW
  "handlerNode":  "ChatService.sendMessage",
  "request": {
    "method": "POST", "path": "/chats/{id}/message",
    "params":  { "id": "chat_42" },
    "headers": { "x-user": "user_b3ad...", "x-trace": "..." },
    "body":    "<FlexBuffer>"             // scrubbed
  },
  "response": { "status": 201, "body": "<FlexBuffer>" },
  "downstream": [
    { "kind": "sql.query",    "target": "supabase", "durationMs": 8,   "rows": 1 },
    { "kind": "r2.put",        "target": "IMAGES",   "durationMs": 42,  "bytes": 81920 },
    { "kind": "bus.publish",   "target": "chat.message.sent", "durationMs": 2 },
    { "kind": "ai.generation", "target": "@cf/meta/llama-3", "durationMs": 340, "tokensIn": 120 }
  ],
  "startedAt": "2026-05-27T18:42:19Z",
  "durationMs": 412,
  "tags":    { "client": "android-1.4.2", "build.sha": "a3c...", "anomaly": null },
  "pinned": false
}

9.3 Sampling

• Default sample rate: 5% of normal traffic. Sufficient to cover the long tail without overflowing R2.
• 100% on error: any request with response.status >= 500 or an uncaught exception is captured regardless of sample.
• 100% on slow: any request whose durationMs exceeds the p99 budget for its endpoint.
• Cohort burst: 100% for users tagged debug.session_id by the dev tools panel — for "trace this user end-to-end" debugging.
• Selector knob: per-endpoint rate overrides in shadow-config.toml so high-value endpoints (auth, payments when added) can be captured at 100%.

10. Shadow stores per surface

10.1 Postgres / Supabase / Cockroach

Mechanism: logical replication. Supabase exposes WAL via the wal2json output plugin. A long-running Worker (or a JVM service in shadow) subscribes to a replication slot named shadow_replica and applies changes to shadow_db on a separate cluster.

• 7-day TTL: every row gets a shadow_inserted_at TIMESTAMPTZ DEFAULT now() column added in the shadow schema (no change to prod schema). Nightly DELETE FROM ... WHERE shadow_inserted_at < now() - interval '7 days' AND id NOT IN (SELECT id FROM shadow_pinned). Vacuum follows.
• Pinned table: shadow_pinned (table_name, row_id, pinned_at, pinned_by, reason). Anomaly detector + manual curator both insert here.
• Per-branch fork: if Neon/Supabase branching is enabled, each dev branch gets a copy-on-write fork of the shadow DB at branch creation. Writes during dev stay in the fork; the underlying shadow continues to refresh from prod.
• Tenant prefix: shadow records get tenant_id = 'shadow_' || original_tenant_id rewritten on replication.

10.2 Memgraph

Mechanism: Memgraph supports DUMP DATABASE for snapshots and triggers for change capture. Hybrid:

• Nightly snapshot: shadow-refresh Worker shells into the prod Memgraph, runs DUMP DATABASE to a Cypherl file, uploads to R2, then a shadow Memgraph instance imports it.
• Delta capture: the ShadowCaptureVisitor also captures Cypher writes (CREATE/MERGE/DELETE) on every request. Between snapshots, deltas can be replayed to bring shadow forward, or to replay one specific write sequence in isolation.
• Tenant scoping: every shadow query is automatically rewritten to add WHERE n.tenant_id STARTS WITH 'shadow_' — protects against accidentally querying prod data.

10.3 R2 IMAGES

Mechanism: read-through caching with copy-on-write writes.

class ShadowR2Binding(
  private val shadow: R2Bucket,         // shadow_IMAGES
  private val production: R2Bucket,     // IMAGES (read-only)
) : R2Bucket {
  override suspend fun get(key: String): R2Object? {
    shadow.get(key)?.let { return it }
    val upstream = production.get(key) ?: return null
    shadow.put(key, upstream.body, metadata = mapOf("shadow.cached" to "true"))
    return upstream
  }
  override suspend fun put(key: String, body: ByteArray) = shadow.put(key, body)
}

10.4 Workers AI

Captures every textGeneration call's input/output. Replay returns the cached response keyed by SHA-256(prompt + model + temperature + maxTokens). Identical inputs always produce the captured output; novel inputs (in dev where prompts change) either fail fast (--strict) or fall through to live Workers AI (--allow-novel).

10.5 Google Sheets / Firebase / FCM / APNs

Stubbed by default. Captured request payloads are logged to a side store so devs can inspect "what would have been sent" without anything actually leaving the shadow cluster. A single shadow Manna sheet in a sandbox Google account can be enabled per-tenant for integration tests that need a real write.

11. The 7-day window + pinned fixtures

11.1 Two-tier retention

ROLLING WINDOW                         PINNED FIXTURES
──────────────────────                 ─────────────────────
moves with wall clock                  permanent
~7 days deep                           no TTL
~50 GB at peak                         grows slowly
auto-trimmed nightly                   human + AI curated

┌─────────────────────────┐            ┌──────────────────────────┐
│ captures from last 7d   │ ─ promote ▶│ pinned (5,000 + entries) │
│ - 5% sample of normal   │ on anomaly │ - "the OOM from Jan"     │
│ - 100% on error/slow    │ / manual   │ - "12k-friend user"      │
│ - 100% for debug.uid    │ pin        │ - "race in chat init"    │
└─────────────────────────┘            └──────────────────────────┘
            │
            ▼
     trim job: 0 4 * * *

11.2 Pin reasons (canonical list)

11.3 Storage maths

Average envelope ~4 KB (compressed FlexBuffer). At 50 rps production traffic × 5% sample = 2.5 captures/sec = 216,000/day = 6.5M/week. At 4 KB each that's ~26 GB for the rolling window. Pinned set assumed to grow to ~50,000 entries over 12 months = ~200 MB. Postgres shadow data depends on retention; a 7-day cut of BestBuds today is ~12 GB. Total shadow R2 + Postgres footprint at maturity: under 50 GB — fits comfortably in a single Hyperdrive-backed Postgres + R2 free tier.

12. Replay engine

12.1 CLI ergonomics

# replay one capture against the local dev worker
shadow replay 01HXKQF8M7XYZ                       # single capture by ID

# replay last hour of /api/feed against branch
shadow replay --endpoint /api/feed --since 1h --against http://localhost:8787

# replay every pinned crash against my branch (regression suite)
shadow replay --pinned --reason crash --against http://localhost:8787

# load-test: replay last 24h at 5x speed
shadow replay --since 24h --speedup 5 --against http://staging-shadow

# diff replay output against captured baseline
shadow replay --since 1h --diff --against http://localhost:8787
# → reports: 2 response body diffs, 1 status diff, 1 P95 regression

12.2 What gets replayed

Replay isn't just "re-execute the request" — it reconstructs the captured causal chain:

1. Set Clock, Random, Uuid to deterministic streams seeded from captureId.
2. Wrap every external binding (R2, Workers AI, Sheets, FCM) so that calls matching the captured downstream return the captured response.
3. Issue the captured request with all original headers preserved (except x-trace, which gets a fresh one tagged shadow.replay-of=<captureId>).
4. Compare the new response + downstream chain to the captured baseline.
5. Report diffs structurally: status mismatch, header diff, body diff (with FlexBuffer-aware path-level reporting), missing downstream call, extra downstream call, latency regression.

12.3 Three replay modes

strict

Response must match the captured baseline byte-for-byte (after FlexBuffer canonicalisation). Used in CI for regression detection.

tolerant

Schema match + value equivalence (numbers within ε, timestamps modulo replay clock, IDs allowed to differ if downstream-consistent). Used during refactors.

exploratory

Diff reported but never fails. Used for "I'm trying something new, just show me what changed." Most-used mode interactively.

13. Stubbing dangerous side effects

The replay engine consults a StubRegistry before any external call leaves the shadow:

object StubRegistry {
  private val stubs = mapOf<CallKind, StubStrategy>(
    CallKind.AI_TEXT_GENERATION  to CachedByPrompt,
    CallKind.SHEETS_APPEND        to NullSink(captureFor = "sheets-writes.jsonl"),
    CallKind.FCM_SEND             to NullSink(captureFor = "fcm-sends.jsonl"),
    CallKind.APNS_SEND            to NullSink(captureFor = "apns-sends.jsonl"),
    CallKind.SUPABASE_AUTH_LINK   to DeterministicSuccess,
    CallKind.WEBHOOK_OUTBOUND     to NullSink(captureFor = "webhooks-out.jsonl"),
  )

  fun strategyFor(call: CallKind): StubStrategy =
    stubs[call] ?: if (call.isIdempotent) PassThrough else FailWithGuidance(call)
}

FailWithGuidance throws an exception with a message like "Call to POST https://api.stripe.com/charges blocked in shadow mode. Add a stub via StubRegistry.register(CallKind.STRIPE_CHARGE, …) or replay with --allow-real-effects." — surfaces the dangerous call instead of silently letting it through.

14. Anomaly detection & auto-pinning

A scheduled Worker runs every 15 min over the rolling capture stream and pins anything that looks unusual:

• Stack-trace clustering — captured 5xx errors with stack traces are hashed (top 10 frames, normalised). A new cluster (or a cluster that hasn't appeared in 30 days) auto-pins one exemplar.
• Latency outliers — per-endpoint p99.9 over a 24h window; any capture exceeding it gets pinned with reason slow_outlier.
• Payload extremes — for each endpoint, track the 99th percentile of payload size. Captures above pin with reason cohort_extreme.
• Schema-edge detection — track which combinations of optional fields appear in request bodies. A combination not seen in the last 30 days auto-pins.
• Causal-chain depth — captures whose downstream.length is above p99 (deep graph fan-out) pin.

Pin budget: anomaly detector pins at most 50 captures per day per category, to keep the pinned set from ballooning. Excess go into a candidate queue for human curation in the Workbench.

15. Privacy & PII scrubbing

Scrubbing happens once at capture time, before anything is written to R2 or shadow Postgres. The shadow store contains no real PII; devs cannot accidentally see it.

Deterministic-faker key: a per-tenant secret HMAC key, scoped only to the capture-time scrubber. Same key for the life of the shadow cluster — so the same user is always "Alice Wonderland" in shadow, every replay. Devs can join across tables.

15.1 Auditing

Every capture's scrubManifest records which fields were scrubbed and by which rule. A nightly compliance job samples 100 random captures, decrypts the manifest, and verifies that no raw PII slipped through (e.g. a free-form text field that wasn't annotated). Failures alert the data-team channel.

16. Tenant isolation

Three layers of "you can't accidentally hit prod":

1. ID prefix — all shadow tenant IDs, user IDs, room IDs are rewritten shadow_. Prod has no such IDs; any query with the prefix returns empty against prod, any query without returns empty against shadow.
2. Bucket / database prefix — separate R2 bucket (shadow_IMAGES), separate Postgres database (shadow_db), separate Memgraph (shadow-memgraph on a sandbox GCP project).
3. Credential isolation — shadow credentials are scoped at the Cloudflare account + GCP project level. The prod service account literally cannot reach shadow infrastructure and vice versa.

17. Time travel

Every capture envelope carries startedAt. Replay can pin to any historical moment within the retention window:

shadow replay --as-of 2026-05-20T14:30:00Z --endpoint /api/feed --since 1h

The engine restores the shadow Postgres + Memgraph to that timestamp via SELECT * FROM ... AT TIMESTAMP (CockroachDB native), or via Postgres logical replay rewind, then runs the requested captures. For points outside the 7-day window, the cold restore from the nightly bestbudsArchiver backup is used (slower, ~10 minutes to materialise).

18. Load testing from real patterns

The capture corpus is also a load-test corpus. Per spec §28.2:

shadow load --since 7d --speedup 10x --target http://staging-shadow --duration 30m

# Internally:
# 1. Sample captures from last 7d, weighted by endpoint frequency in prod.
# 2. Replay them at 10x the original arrival cadence.
# 3. k6 collects per-endpoint latency histograms, errors, throughput.
# 4. Export to Mimir; Grafana panels render against the performance-observability
#    pipeline (https://reaktor.build/docs/performance-observability).

Pattern fidelity: traffic is replayed with the same inter-arrival distribution as production — bursts, idle periods, diurnal cycles — not as a uniform RPS stream. This catches issues that uniform load tests miss (queue depth spikes, cache stampedes).

19. Observability

Every capture and replay emits OTel spans into the same Grafana LGTM pipeline described in performance-observability §17. Two service names:

• service.name = "shadow-capture" — the visitor itself. Tells you capture overhead, sample drops, scrubber failures.
• service.name = "shadow-replay" — every replay run. Tells you replay duration, diffs detected, false-positive rate.

The Workbench gets a Shadow tab showing: captures/sec live, pinned set size, oldest unpinned capture, today's replay diff count, anomaly auto-pin rate.

20. Workbench & CLI

20.1 Workbench panel

A dedicated Shadow panel in the Reaktor Desktop workbench (lives alongside the existing Blueprint editor):

• Capture stream — live tail of incoming captures, filter by endpoint / tenant / status. Click → inspect envelope.
• Pinned fixtures — browsable, searchable, pinnable/unpinnable. Each entry shows: pin reason, age, last replayed, recent diff count.
• Replay against branch — pick any capture, hit "Replay against my branch" → opens a diff view (response, downstream, latency).
• Anomaly candidates — the queue the auto-pinner couldn't keep up with; human-curator approves or dismisses.
• Refresh status — when did Postgres replication last lag? When did Memgraph last snapshot? When did the trim job last run?

20.2 CLI verbs

21. CI integration

Three CI jobs per PR, all running against the shadow stack:

CI · job 1~2 min

Replay pinned regression suite

shadow replay --pinned --strict --against $PR_BRANCH. Every pinned crash and incident-repro fixture must still produce the captured response. Failure blocks merge.

CI · job 2~3 min

Replay yesterday's traffic sample

shadow replay --since 24h --sample 1% --tolerant. A random 1% of the last day's captures replayed in tolerant mode. Catches behavioural drift that pinned fixtures don't cover yet.

CI · job 3~10 min (nightly)

Load test from last week's patterns

shadow load --since 7d --speedup 5x --duration 8m. Runs nightly on main, not per-PR. Latency histograms feed the perf dashboard; regressions vs the previous nightly create a ticket.

22. BestBuds wiring

22.1 First slice: chatServer

Start with one worker — chatServer — because it has the richest traffic and an obvious high-value debug surface. Validate the capture path on one worker before fanning out to the other nine.

23. Phased plan

Phase 0~3 days

Spec the envelope & pin schema

CaptureEnvelope, PinReason, StubStrategy, ShadowConfig in reaktor-shadow commonMain. FlexBuffer schemas + serde. Unit tests on round-trip.

Phase 1~1 week

Capture path on one worker

ShadowCaptureVisitor in TS for chatServer. Captures envelopes to shadow_captures R2 bucket. PII scrubber for chat-message fields. 5% sampling. Done when: 24h of chatServer traffic visible in R2 as well-formed envelopes, <1% capture overhead per request.

Phase 2~1 week

Shadow Postgres via logical replication

Provision shadow Supabase project. Set up wal2json replication slot. Subscriber Worker applies changes to shadow with tenant prefix rewrite + shadow_inserted_at column. Trim job. Done when: shadow DB lags prod by < 5 s and 7-day trim works.

Phase 3~1 week

Shadow R2 + Memgraph + stubs

Read-through R2 binding. Nightly Memgraph dump + import. Workers AI cache stub. Sheets / FCM null sinks. Done when: a dev can run chatServer locally against the full shadow stack and observe end-to-end requests producing no real side effects.

Phase 4~1 week

Replay engine + CLI

The shadow CLI with replay, inspect, pin, diff. Three replay modes (strict/tolerant/exploratory). Diff reporter. Done when: a captured request from yesterday can be replayed against the local branch and produces a structured diff.

Phase 5~½ week

Anomaly detector + pin queue

Scheduled 15-min Worker doing stack-trace clustering + latency outliers + payload extremes. Pin budget (50/cat/day). Candidate queue surfaces in Workbench.

Phase 6~1 week

Roll capture to all 10 workers

Apply the Phase-1 middleware to every worker. Tune per-endpoint sample rates. Validate scrubber coverage on every endpoint's schema.

Phase 7~½ week

CI integration

Three CI jobs above. PR-comment formatting for diff reports.

Phase 8~1 week

Load test + Workbench Shadow panel

shadow load verb + k6 integration. The Workbench Shadow tab. Grafana dashboards for capture/replay metrics.

Phase 9~1 week

Time travel + per-branch DB forks

shadow restore --as-of. Supabase branching for per-PR DB forks (only if Supabase enables it on the account tier).

24. Risks & open questions

Risks

• Capture overhead in prod. The visitor sits in every request path. Target: <1% latency overhead at p95 with 5% sampling. Mitigation: capture envelope assembly is async, writes batched to R2 every 200 ms, dropped on backpressure.
• Logical replication lag. If shadow DB lags prod by minutes, recent captures replay against stale data. Mitigation: monitor replication slot lag as a first-class metric; alert if > 30 s.
• R2 cost. 26 GB rolling + 200 MB pinned at $0.015/GB/mo = trivial. Class A writes at 216k/day × $0.0045 per 1k = ~$0.03/day. Capture egress for replay is the wild card — if devs replay aggressively, bandwidth costs spike. Mitigation: replay reads stay within Cloudflare's network (free).
• PII scrubber misses a field. A new schema field added without @Pii annotation could leak real PII into shadow. Mitigation: nightly compliance sampler (§15.1) + a schema-evolution CI check that flags any new free-form text field without an explicit PII annotation.
• Memgraph snapshot freshness. Nightly is too stale for some bugs. Mitigation: the delta-capture stream allows fine-grained roll-forward; the snapshot interval can shrink to 6h if needed.
• Stub registry stales. When prod adds a new external API, the stub registry doesn't know about it — replay either fails or fires the real call. Mitigation: FailWithGuidance default + a CI check that scans Kotlin/TS for new HTTP-client construction.
• Replay non-determinism. A handler that reads system time, RNG, or hostname without going through the framework will not be deterministic. Mitigation: a BlockHound-style runtime check (in CI only) that catches System.currentTimeMillis, UUID.randomUUID, etc. outside the framework's clock / id facilities.
• Per-branch DB fork cost. If we go the Supabase-branching route, every PR creates a DB fork. Mitigation: only branch on demand (when a dev explicitly runs shadow branch create), prune after 7 days of inactivity.

Open questions

• Does the shadow stack live in a separate Cloudflare account, or same-account-different-zone? Separate is safer (isolation), same is cheaper (no inter-account billing).
• Memgraph CE on GCP has a per-instance cost; do we run one shadow Memgraph 24/7 or spin one up on demand for replay sessions? On-demand is cheaper but adds 2-min startup latency to replay.
• Capture corpus retention beyond 7 days for pinned + 30 days for stack-trace fingerprints — is that GDPR-safe given the scrubber? Likely yes (no real PII), but legal sign-off needed before public launch.
• Do we expose shadow access to external contributors (if Reaktor is open-sourced)? Probably no — captures still encode tenant-specific shapes that could be reverse-engineered.
• Integration with the existing bestbudsArchiver: should the archiver's nightly Postgres dump become the cold-restore source for shadow, or is that a separate path?
• How does this compose with reaktor-notifications' outbox pattern when that lands? The outbox row IS already a perfect capture; we may be able to bootstrap from outbox content rather than installing a separate visitor.

25. References

Spec sections this design extends

• §28 Reaktor Shadow: Testing & Quality Infrastructure — the "capture as a visitor" foundation this doc operationalises into a permanent dev environment.
• §21.8 Transactional outbox — the pattern that lets us safely capture publish intentions without firing them in shadow.
• §21.9 Idempotent consumers — replay correctness depends on this; the bus design assumes consumers handle duplicates.
• §29 Performance: Measurement, Profiling, and Optimization — load testing & OTel pipeline this doc plugs into.
• §29.4 Automated performance testing — Keploy / k6 / Maestro integration assumed.

Peer designs in this repo

• performance-observability — Grafana LGTM pipeline shadow capture/replay flows into; Keploy + k6 driver runners.
• reaktor-notifications-design — uses outbox; FCM/APNs stubs from §13 reuse the same null-sink pattern.
• deep-link-integration — same graph-visitor pattern (RouteIndexVisitor, BackStackPlannerVisitor); a deep-link tap is a typical capture target.
• auth-analysis — service principals + tenant API keys; shadow runs as a dedicated service principal.
• security-plan — PII scrubbing rules; shadow extends them with the deterministic-faker key.
• graph-kernel-analysis — visitor + facets foundation.
• reaktor-graph-visitors-design — base classes ShadowCaptureVisitor extends.

Files inventoried during the design

• bestbuds/wrangler.json (root) — Hyperdrive bindings: supabase, supabasedirect, cockroach, cockroachdirect; R2: IMAGES.
• bestbuds/targets/{analytics,chat,sticker,config,bot,bestbudsArchiver,media,notification,social,messaging,prototype}{Server,Worker}/wrangler.json — 10 production workers + 1 archiver + 1 prototype.
• bestbuds/targets/reaktorServer/src/main/kotlin/ai/bestbuds/server/ServerApplication.kt — Spring Boot REST + GraphQL surface, memgraphDriver() on lines 18, 294.
• bestbuds/targets/reaktorServer/.../MannaKnowledgeGraphGraphQl.kt — Cypher queries; node labels (Domain, Resource, Chapter, Tenant, Task) and edges (ChapterEdge, TaskDependencyEdge, TaskResourceEdge, GoalTaskEdge).
• bestbuds/modules/app/src/jvmMain/kotlin/ai/bestbuds/app/db/schema.sql — Postgres schema (user_profiles, friends, groups, group_users, embeddings, heimdall.personal_access_token).
• bestbuds/n8n/workflows/ — 10 deploy / build / sync workflows; none for shadow refresh yet.

External tools cited

• Keploy — capture/replay regression testing
• GoReplay — TCP traffic mirroring
• mirrord — local-process-in-remote-pod
• Replay.io — deterministic record/replay
• Supabase Branching
• Neon Branching
• Memgraph DUMP DATABASE
• Supabase logical replication
• Cloudflare R2 event notifications
• k6 load testing

Doc lineage. Authored 2026-05-28. Grounded in Reaktor Spec §28 (the visitor-based capture primitive) and a fresh inventory of BestBuds' actual production data surfaces (10 Cloudflare workers, 4 Hyperdrive bindings, 2 R2 buckets, 1 JVM service, 1 Memgraph on GCP CE, plus Workers AI + Google Sheets + planned FCM/APNs). Cross-linked to the deep-link, performance, and notifications design briefs published this week.

Related docs

• Shadow Dev - Use before building production-like test and replay environments.
• Codex Dev Env - Use only when comparing prior shadow environment proposals.
• Perf and Observability - Use before optimizing or adding instrumentation.
• Reaktor Secrets - Use before moving, rotating, distributing, caching, or materializing secrets.
• DevOps - Use before changing the build/deploy/ops pipeline, the Kotlin platform model, deployment partitioning, gated deploy operations, or the reaktorDesktop Deploy pane; pairs with reaktor-cloud-pane (operator view) and secrets-management.
• FlexBuffer Plan - Use for history only; prefer the FlexBuffer Binary Serialization document.