Shadow design history

Superseded design archive, preserved 1 October 2026. The text below is the pre-rewrite document, including its dated reviews. Its current-state claims, schedules, isolation assumptions and guarantees may be obsolete. Read Operations & Reliability and the current overview for the reconciled direction.

Two environments only: dev and prod. Build §0 · MVP Dev Environment first — the concrete, buildable dev/prod separation. Everything from §1 on is the "shadow" north star the MVP grows into; per Principle #2 (§6), the shadow is dev, so the tier you stand up in §0 becomes the substrate the shadow later populates. Nothing in §0 is throwaway.

0. MVP Dev Environment — build this first

Goal: a developer (and CI, and the Maestro device farm) runs a real dev build against a real, isolated dev backend — separate data, separate auth identity — that never touches prod, while a dev and a prod app coexist on the same phone. This is the minimum that makes the shadow machinery (§1+) buildable, and it is everything needed to test features like message-notifications end-to-end without risking prod.

Two tiers only: dev and prod. No stage.

0.1 Production-backed development

• Environment ships { STAGE, PROD }, but STAGE is dead: ExposedAdapter(stageDb = prodDb = supabaseDb()) (targets/reaktorServer/.../ServerApplication.kt:410) — both handles point at the one Supabase.
• The app hardcodes prod everywhere: Environment.PROD, the 13 Servers.* URLs, BESTBUDS_APP_ID = 38d1cee3…, Firebase project mehmaan-app, the OAuth client ids, applicationId = ai.bestbuds.app. No product flavors, no buildConfig, no server-switcher.
• All 20 workers share one Cloudflare account, one Hyperdrive → one Supabase, one D1, one R2 set — zero [env] sections.
• Auth is one realm (api.reaktor.build); issuer/JWKS/app-id are compile-time constants; dev-login is local-only impersonation that cannot authenticate against the workers (UserRepository.impersonate sets no Feature.Auth session).
• The worker-service delegation is mis-provisioned (403 on chat.socket), so the realtime socket + message-push are broken on prod anyway.

0.2 Phase 0 — three quick wins (independent; do immediately)

Not "dev environment" work — prod fixes worth landing first.

1. Grant the worker-service delegation. scripts/provision-worker-service-auth.mjs provisions bestbuds-worker-service with allowed_audiences=[app-id], scopes="worker:server" — but the chat socket exchanges for audience="chat.socket", scope="realtime:connect", which fails the audienceAllowed gate (reaktor-auth/.../AuthTokenGrantNode.tokenExchange → 403). Fix: add chat.socket (+ any realtime audiences) to allowed_audiences and realtime:connect auth:token-exchange notification:dispatch messaging:read to scopes — the set the karate fixture (tests/reaktorServer/karate/seed-auth-fixtures.mjs) already uses — then re-provision. This alone resurrects the chat socket and the message-notification push on prod.
2. Rotate committed secrets. Move them to the secret store and rotate them; the list is tracked privately.
3. Close the database exposure tracked privately. Fix it before any shared-database decision; it is a standalone production exposure.

0.3 The spine — one Environment, resolved at each composition root

Collapse the enum to the two tiers you actually run:

// reaktor-service/src/commonMain/.../service/util.kt
enum class Environment { DEV, PROD;  companion object {
  val Header = "X-Environment"
  operator fun invoke(v: String) = if (v.lowercase() == "prod") PROD else DEV
}}

Rename the dead STAGE → DEV; it already drives the X-Environment header and ExposedAdapter's DB pick. Each layer reads its env-dependent values from a data object populated once at its composition root (the reaktor rule: never getenv inside a node). One value, threaded outward:

0.4 Build it — four moves

A · Client — an AppEnvironment object + a build variant. Introduce one data object holding every env-coupled value (the 13 Servers URLs incl. ConfigServer, the app-id UUID, OAuth client ids, fcmProjectId, the Environment header). Provide AppEnvironment.Prod/.Dev; apply it before Reaktor.start (service singletons capture Servers.* at init). Replace the scattered prod constants (38d1cee3, OAuth ids, mehmaan-app, Environment.PROD, the Servers defaults) with reads from it.

• Android: add flavorDimensions("env") + dev/prod product flavors in targets/appAndroid/build.gradle.kts. dev gets applicationIdSuffix ".dev" (dev+prod coexist on one device — keep namespace = ai.bestbuds.app stable), a src/dev/google-services.json, and a BuildConfig.FLAVOR the entrypoint reads to pick the AppEnvironment. Enable buildFeatures.buildConfig (the dependeasy defaults() doesn't today).
• iOS: add a Debug-Dev config + .xcconfig (or a duplicated iosApp-Dev scheme) setting PRODUCT_BUNDLE_IDENTIFIER = ai.bestbuds.app.dev, pointing at a dev GoogleService-Info.plist, and defining a DEV compile flag the entrypoint keys off.
• Keep the remote ConfigServer → Servers.applyConfig indirection as a secondary layer on top of the chosen env's config worker (per-env flags still work); the base env is fixed by the build.

B · Workers — render per-env config, add a TARGET_ENV knob. Two committed files per worker — wrangler.prod.json + wrangler.dev.json (differ only in name / resource ids / routes / vars) — plus per-env root files, rendered from a new environments map in the reaktor key of package.json (the one convention point the CLI, Dagger, and merge all respect). Deploy selects via TARGET_ENV=dev in the deployX scripts + Dagger deployWorkers(env). Prefer this over wrangler [env.dev] sections — the predeploy jq-merge and the 6 no-predeploy workers fight wrangler's env inheritance. Dev fleet = -dev names, dev bindings, service-binding targets repointed to -dev peers. (Fix first: the inconsistent root-merge order across workers, and the stale shibasis0801-chat-service / app-service binding names — or dev silently binds to prod.) Dev secrets via the existing WORKER_SERVICE_SECRET_TARGETS / CLICKHOUSE_SECRET_TARGETS overrides in the secret scripts.

C · Data — separate the cheap stores, namespace the expensive ones.

D · Auth — un-collapse the DB, give dev its own app-id, make dev-login real.

• Point ExposedAdapter.stageDb → the dev Supabase (un-collapse ServerApplication.kt:410); Environment.DEV now routes to the dev DB. Keep one reaktor-auth deploy for the MVP.
• Register a dev app-id (audience) UUID in the dev DB; thread it through AppEnvironment (replacing the hardcoded 38d1cee3 on the client + BESTBUDS_APP_ID on the workers). Same issuer/JWKS is fine for the MVP — the app-id/audience is the isolation boundary; a separate dev.api.reaktor.build realm is a later hardening step.
• Provision a dev worker-service client (with the delegation grants from §0.2.1) against the dev DB; push its secret to the -dev workers.
• Real dev-login: generalize scripts/seed-dev-auth-fixtures.mjs to seed sessions + mint tokens against the dev realm (it already seeds heimdall.session/refresh_token and calls /auth/session/refresh). This replaces local-only impersonate() with genuine dev tokens, so dev builds actually authenticate. Keep DevAuthServerInterceptor + ephemeral signing keys gated by real env — never enabled on prod.

0.5 Deploy — dev vs prod

See DEPLOY.md for the full checklist; the two-env delta:

# prod (unchanged)
./gradlew :app:jsBrowserProductionLibraryDistribution
TARGET_ENV=prod npm run deployChat        # → chat-service           (wrangler.prod.json)

# dev
./gradlew :app:jsBrowserProductionLibraryDistribution
TARGET_ENV=dev  npm run deployChat        # → chat-service-dev       (wrangler.dev.json)

Service-binding deploy order is identical per env (text-moderator → image-moderator → chat → messaging → media → campaign-engagement). Client: the Android dev flavor (assembleDevDebug) / the iOS iosApp-Dev scheme.

0.6 MVP checklist (ordered)

1. Phase 0 — grant delegation, rotate secrets, close the database exposure.
2. Environment { DEV, PROD }; AppEnvironment.{Dev,Prod}; replace the hardcoded constants.
3. Android dev/prod flavors (+ .dev suffix, src/dev/google-services.json); iOS Dev scheme/xcconfig.
4. Provision dev data — Supabase+Hyperdrive, D1, R2 buckets, graph dev tenant, bestbuds_dev ClickHouse.
5. reaktor.environments block → render wrangler.{prod,dev}.json; add TARGET_ENV to deployX + Dagger.
6. Un-collapse stageDb → dev; register dev app-id; provision dev worker-service (with grants); set dev secrets.
7. Real dev-login via dev seed-fixtures.
8. Verify: install ai.bestbuds.app.dev alongside prod; dev-login; send a message dev→dev with the receiver backgrounded; confirm the push + tap-opens-chat (the exact flow that couldn't be completed on prod because of the §0.2.1 delegation 403).

0.7 How the MVP grows into the shadow (§1+)

The MVP gives dev its own isolated stores + auth + build. The shadow layers on top: the dev Supabase/D1/R2 become the targets the materialization loop (§16) hydrates from captured prod envelopes; the dev worker fleet gains ShadowCaptureMiddleware (§11.5); "dev data" stops being hand-seeded and becomes a rolling 7-day mirror of prod (§15). §0 is the substrate; §1+ populates it.

Dev runs against a continuously-refreshed shadow of production: the latest 7 days of Supabase rows, Memgraph nodes, R2 objects, worker payloads, and JVM API traces — plus a hand-curated library of pinned crashes and anomaly capsules that never expire. All testing, load testing, and exploratory debugging happens here. Built as a graph visitor (per Reaktor Spec §28) extended end-to-end across the actual data surfaces in BestBuds.

• Builds on — reaktor-shadow

• First consumer BestBuds dev cluster

• Window rolling 7-day + permanent pinned capsules

• Test stack — Playwright · Maestro · Karate · k6 · Keploy · Shadow Worker Proxy

• Capture front-ends — 2 (Keploy @ k3s/JVM · Shadow Worker Proxy @ Cloudflare edge) → one R2 corpus

• Edge capture surface — 5 layers (entropy/clock oracle · all entrypoints · generic env-proxy · req/res internals · out-of-band tail + Logpush)

• Replay frontier — 8 capabilities (omniscient debug · version/runtime diff · live shadow · fuzz + fault · distributed DAG · stateful DO · self-curating corpus · encrypted capture)

• Doc merged from Codex + Claude drafts (2026-05-28); edge proxy + full test stack added 2026-06-05

• Stateful surfaces — 7 (Postgres · Memgraph · R2 × 2 · Workers AI · Sheets · JVM)

• Cloudflare workers in scope — 10

• Storage budget — ~50 GB at peak

• Refresh cadence — continuous capture · nightly trim

• PII handling — redact at capture (fail-closed)

• Side-effect safety — tenant prefix + stub registry

• Environment classes — 4 (current · pinned · pr-<id> · local-<user>)

• Capsule classes — 5 (Rolling · Latency · Crash · Anomaly · Incident)

1. Executive summary

"Dev" today means an empty database and a stubbed worker — production bugs lived in a state of the world that empty dev environment never produced. The shadow dev environment fixes this by making production state the default state of dev: the same Supabase tables, the same Memgraph subgraphs, the same R2 objects, the same chat-message shapes, all updated continuously and trimmed to a rolling 7-day window. Any captured trace can be replayed against your branch — like Keploy, but covering all six Reaktor transports (HTTP · LOCAL · PEER · PUBSUB · QUEUE · WORKFLOW), every datastore, and the full causal chain of downstream calls a request triggers.

The right abstraction is reaktor-shadow: an application-layer capture, materialization, and replay system. Keploy remains a useful runner for HTTP-style regression tests; Reaktor captures one layer above so a single trace can include a Worker request, service handler, D1 / Postgres query, Durable Object message, R2 object reference, Memgraph subgraph, queue event, and downstream server response — together.

The architecture has three loops:

• Capture loop — a ShadowCaptureVisitor attaches at every ServiceNode, RepositoryNode, ActorNode, and WorkerBase. Every request/response pair, every DB write, every external call is bundled into a ShadowEnvelope with explicit resources[] (refs to the Supabase rows, R2 keys, DO state, Memgraph subgraph it touched). Envelopes go to R2; metadata goes to a catalog index; PII is redacted before either.
• Materialization loop — nightly + on-demand jobs walk envelope refs, compute resource closure (FK closure on Postgres, subgraph closure on Memgraph, object closure on R2), redact, and hydrate the immutable shadow-current base. Pinned capsules apply on top.
• Replay loop — devs run shadow replay <id> (CLI, Workbench button, CI job). The engine reconstructs the request context, deflects external API calls to deterministic stubs, executes against the branch, compares per-policy (strict / shape-stable / statistical / side-effect), and reports diffs.

2. Current state

2.1 What dev/test infrastructure exists today

2.2 The honest gap

Today a dev with a branch has three options, all bad: (a) run nothing, write code blind; (b) wire their laptop to prod and hope they don't fat-finger a destructive query; (c) spin up an empty local Postgres + Memgraph + Workers and hand-craft fixtures that never resemble production. Option (c) is what most teams settle for, and it's why every "it worked locally" bug exists. (§0 replaces all three with a real, isolated dev tier.)

2.3 First implemented slice: Keploy + R2 for reaktorServer

The first concrete reaktor-shadow slice is intentionally narrower than the full design: capture HTTP traffic for the k3s app-server Deployment, archive the generated Keploy bundle to Cloudflare R2, pull that bundle onto a developer machine, and replay it locally with the Keploy CLI.

# one-time cluster install, after creating the cluster entry in Keploy
KEPLOY_ACCESS_KEY=... npm run keploy:install

# capture live k3s traffic for reaktor/app-server
KEPLOY_PAT=... npm run keploy:record:start
KEPLOY_PAT=... npm run keploy:record:status

# archive the active bundle to R2, then stop recording
KEPLOY_PAT=... \
R2_ACCOUNT_ID=... R2_BUCKET=reaktor-keploy \
AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... \
npm run keploy:record:stop

# on the local machine: pull latest bundle and replay
R2_ACCOUNT_ID=... R2_BUCKET=reaktor-keploy \
AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... \
npm run keploy:r2:pull
npm run keploy:test

The scripts live in two places: cloud/k3s/*keploy*.sh owns cluster install, record, status, stop, and R2 upload; targets/reaktorServer/keploy-r2-pull.sh and targets/reaktorServer/keploy-local-test.sh own local replay. The local runner defaults to the same app-server image used by k3s, but KEPLOY_COMMAND can override it for a local Spring Boot command or a Docker Compose stack.

This is the HTTP adapter, not full shadow replay. It does not yet compute Postgres/Memgraph/R2 dependency closure, does not stub dangerous side effects, and does not attach graph node attribution. Its value is that recorded production-shaped API behavior now becomes a portable, durable test input instead of disappearing with the cluster session.

2.4 Second slice: the edge gap (this design)

The k3s adapter covers the JVM tier and nothing else. Every one of the 10 BestBuds Workers is invisible to it, and Keploy can never reach them: a V8 isolate has no kernel and no sidecar for eBPF or k8s-proxy to attach to. This doc adds the Shadow Worker Proxy — an in-isolate capture/replay front-end that gives the Cloudflare edge the same record → R2 → replay → diff loop Keploy gives the cluster, writing into the same R2 corpus. See §11.4–§11.5 for capture and §17.4 for replay; §17.5 places both inside the full Playwright · Maestro · Karate · k6 stack.

3. Why a moving shadow

Bugs live in production shapes

The user with 4,200 chat messages, the group with 87 members, the friend graph with a 6-hop cycle — these don't appear in fixtures. Reproducing requires the actual data; recent actual data, because schemas and content drift week to week.

Load tests need realistic mix

k6 scripts that hit /api/feed at uniform 100 rps don't predict the production failure mode where one user with 12,000 friends makes a single read that joins 12,000 rows. Real traffic patterns are the input; synthetic load is the multiplier.

Edge cases need permanence

A crash from January that took 6 hours to root-cause must be reproducible in October — not "if you can find the row in the backup." Pinned capsules keep the weird ones forever.

Side effects need a sandbox

Devs experimenting can't be allowed to send real FCM notifications, write to the real Manna Google Sheet, or trigger Workers AI charges. Shadow stack means side effects are intercepted at the visitor.

Privacy by construction

PII redacted once at capture time beats redacting on every query. Shadow data is already safe to look at; devs don't need to remember to anonymise.

Replay closes the regression loop

Every captured trace is a candidate test case. CI replays the last week of distinct traffic against the PR branch. If the response shape or P95 latency drifts, the PR is flagged.

4. Spec alignment

The Reaktor spec places reaktor-shadow in the tooling layer and intelligence/tooling plane. It calls for application-layer production traffic capture, local/CI replay, graph-community-scoped regression matrices, visual snapshot diffing, Keploy integration, and k6/Gatling load generation from captured traffic patterns.

5. Prior art

6. Principles

1 · Capture is a visitor, not a proxy

Per spec §28.1: ShadowCaptureVisitor attaches at the graph layer. Every transport, every node, one mechanism. No sidecar, no service mesh, no eBPF.

2 · The shadow IS dev

There is no separate "dev environment". The shadow stack is dev. Devs run their branch against shadow workers, shadow Postgres, shadow Memgraph, shadow R2.

3 · Moving by default, pinned by exception

Default is a 7-day rolling window — captures older than that are trimmed. Exceptions (crashes, anomalies, hand-curated edge cases) are pinned and never expire.

4 · Base + overlay everywhere

Each stateful surface has an immutable base (the moving shadow) and a writable overlay (per-PR, per-developer). Devs only write to overlay; the base resets daily.

5 · Side effects fail loudly in shadow

Every external call (FCM, APNs, Workers AI, Google Sheets) hits a stub by default. Bypassing requires explicit --allow-real-effects + a confirmation prompt.

6 · Redaction is fail-closed at capture

Unknown fields are dropped, not preserved. Schema annotations + a denylist. The shadow store contains no real PII.

7 · Replay must be deterministic

Time, RNG, IDs, external calls — all read from the captured envelope, not the system. A replay that gives a different answer means the code changed, not the environment.

8 · Production-shaped, not production-identical

The goal is preserving behaviour, topology, cardinality, route mix, failure modes — while stripping identity, secrets, unnecessary private payloads.

7. Stateful surfaces to shadow

From a fresh audit of bestbuds/targets/*Server, cloud/, and the JVM reaktorServer:

8. Environment classes

Four flavours of shadow environment, each with a different mutation policy and lifetime. All share the same capture stream and capsule catalog:

8.1 Base + overlay layout per resource

shadow-current-base-2026-05-28          ← immutable, rebuilt daily
  supabase_base schema                  sanitised rows from prod replication
  cockroach_base                        CDC mirror
  r2_base bucket prefix                 copied / transformed objects
  memgraph_base                         imported subgraph from DUMP DATABASE
  do_base snapshots                     (future) per-class DO snapshot refs

shadow-local-ovd                        ← writable overlay, devs mutate here
  supabase_overlay schema               developer writes
  cockroach_overlay                     developer writes
  r2_overlay prefix                     new objects
  memgraph_overlay                      local changes
  do_overlay namespace                  actor state mutations

9. Side-effect classification

Replay correctness hinges on knowing which calls are safe to re-execute (idempotent reads, deterministic computations) vs which would produce duplicate real-world effects. Three categories, each with a default behaviour:

The visitor classifies on capture using two signals: HttpMethod for HTTP (GET/HEAD/OPTIONS → idempotent), and explicit @SideEffect(idempotent = true|false) annotations on Reaktor service handlers. Handlers without an annotation default to non-idempotent (safe default — better to over-stub than over-fire).

10. Stack diagram

┌────────────────────────── PRODUCTION ────────────────────────────────┐
│ user request                                                         │
│     │                                                                │
│     ▼                                                                │
│ ┌────────────────────────────────────────────────────────────────┐   │
│ │ Cloudflare Worker / JVM Service                               │   │
│ │ ┌────────────────────────────────────────────────────────────┐ │   │
│ │ │ ServiceNode                                                │ │   │
│ │ │ ├─ ShadowCaptureVisitor.intercept(req)                     │ │   │
│ │ │ │  ↓ writes envelope → R2 captures bucket                  │ │   │
│ │ │ │  ↓ writes catalog row → D1/Postgres index                │ │   │
│ │ │ ├─ RepositoryNode → Postgres (logical replication)          │─┼──┐
│ │ │ ├─ GraphNode → Memgraph (CDC + nightly dump)                │ │  │
│ │ │ ├─ ActorNode → bus publish (outbox)                         │ │  │
│ │ │ └─ ExternalCall → FCM/Sheets/AI (captured)                  │ │  │
│ │ └────────────────────────────────────────────────────────────┘ │   │
│ └────────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────│────────────────────────────────────┘
                                  │ replication · capture · closure
                                  ▼
┌────────────────────────── SHADOW LAKE ──────────────────────────────────┐
│ R2 raw bodies + trace artifacts · D1/Postgres catalog index ·           │
│ pinned capsule registry · graph/community labels (reaktor-graphify)     │
└────────────────────────────────────────────────────────┬────────────────┘
                                                         │
                                                         ▼
┌────────────────────────── SHADOW ENVIRONMENTS ──────────────────────────┐
│ ┌────────────────────┐ ┌────────────────────┐ ┌────────────────────┐   │
│ │ shadow-current     │ │ shadow-pinned/*    │ │ shadow-pr-<id>     │   │
│ │ rolling 7d         │ │ incident capsules  │ │ ephemeral CI       │   │
│ │ rebuilt daily      │ │ 180d + manual      │ │ destroyed on close │   │
│ └────────────────────┘ └────────────────────┘ └────────────────────┘   │
│ ┌──────────────────────────────────────────────────────────────────────┐ │
│ │ Shadow workers (mirror of prod fleet × 10)                           │ │
│ │ analytics · chat · sticker · config · media · notification ·         │ │
│ │ social · messaging · bot · prototype — all with shadow bindings      │ │
│ └──────────────────────────────────────────────────────────────────────┘ │
│                     ▲                                                    │
│                     │ dev branch / CI replay                             │
│ ┌───────────────────┴──────────────────────────────────────────────────┐ │
│ │ ShadowReplay engine · CLI · CI job · Workbench panel                 │ │
│ │ Keploy adapter · Reaktor service replay · DO replay · k6             │ │
│ └──────────────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────┘
          ▲
          │ nightly trim · anomaly auto-pin · materialize
┌───────────────────────────┐
│ ShadowRefresh Worker      │
│ cron: 0 4 * * * UTC       │
└───────────────────────────┘

11. Capture plane

Capture happens at the same typed boundary the app already uses: Reaktor service handlers, graph actions, repository calls, bus envelopes, and actor/worker messages. Raw HTTP capture is not enough because BestBuds spans Cloudflare Workers, future Durable Objects, queues, R2, Supabase/Postgres, Memgraph, and JVM services.

11.1 The visitor

class ShadowCaptureVisitor(
  private val sink: CaptureSink,
  private val redactor: RedactionPolicy,
  private val sampler: CaptureSampler,
  private val catalog: CaptureCatalog,
) : Visitor() {

  override fun visitServiceNode(node: ServiceNode): ExitScope {
    node.intercept(object : ServiceInterceptor {
      override suspend fun intercept(req: Request, chain: ServiceChain): Response {
        if (!sampler.shouldCapture(req)) return chain.proceed(req)

        val captureId = Uuid.v7().toString()
        val resources = mutableListOf<ShadowResourceRef>()

        // Push CaptureContext into coroutine context so child service / DB /
        // actor calls all append to `resources`.
        return withCaptureContext(CaptureContext(captureId, resources)) {
          val startedAt = Clock.System.now()
          val response = try { chain.proceed(req) }
                         catch (t: Throwable) { sink.captureError(captureId, t); throw t }
          val finishedAt = Clock.System.now()

          val envelope = ShadowEnvelope(
            schemaVersion = "reaktor.shadow.v1",
            captureId     = captureId,
            tenantId      = req.tenantId,
            transport     = req.transport,
            route         = RouteRef(req.method, req.pattern),
            graphNode     = node.id,
            request       = redactor.redact(req),
            response      = redactor.redact(response),
            resources     = resources.toList(),
            timing        = Timing(durationMs = (finishedAt - startedAt).inWholeMilliseconds),
            startedAt     = startedAt,
          )
          sink.write(envelope)
          catalog.index(envelope)            // queryable by time, route, community, anomalyScore
          response
        }
      }
    })
    return NoOpExit
  }

  override fun visitRepositoryNode(node: RepositoryNode): ExitScope { /* append SQL refs */ }
  override fun visitActorNode(node: ActorNode):           ExitScope { /* append DO/actor refs */ }
}

11.2 Capture adapters per surface

11.3 Sampling

• Default sample rate: 5% of normal traffic. Sufficient to cover the long tail without overflowing R2.
• 100% on error: any request with response.status >= 500 or uncaught exception is captured regardless of sample.
• 100% on slow: any request exceeding endpoint p99 budget.
• Cohort burst: 100% for users tagged debug.session_id — "trace this user end-to-end" debugging.
• Per-endpoint override: high-value endpoints (auth, payments when added) capture at 100% via shadow-config.toml.

11.4 Edge capture: the Cloudflare Workers proxy (Keploy-equivalent)

§2.3's first slice records the JVM/k3s tier: Keploy's k8s-proxy taps app-server and archives bundles to R2. That mechanism stops dead at the cluster boundary. The 10 BestBuds Workers run on Cloudflare's edge, and Keploy cannot capture a single request there — by construction:

• Keploy records by attaching an eBPF probe in the Linux kernel or a sidecar / k8s-proxy container that intercepts the app container's socket syscalls. Both need a kernel and a pod.
• A Cloudflare Worker is a V8 isolate, not a container: no kernel access, no sidecar, no privileged network namespace, no process to attach eBPF to. There is nothing for Keploy to hook.

So the edge needs its own capture/replay front-end with the same record → R2 → replay → diff contract as Keploy, realised entirely at the application layer inside the isolate. We call it the Shadow Worker Proxy.

Keploy records two halves of every interaction — the inbound request/response, and the outbound mocks (every dependency call the handler made, so replay can stub them). The Shadow Worker Proxy reproduces both halves from inside the isolate, using three composable mechanisms:

┌────────────────────────── CLOUDFLARE EDGE ───────────────────────────────┐
│  client ─▶ Worker (V8 isolate)                                            │
│              │   ┌─────────────────────────────────────────────────────┐ │
│              │   │ ShadowCaptureMiddleware  (wraps Service.toHono)      │ │
│              ├──▶│  • clone request  • tee response                     │ │
│              │   │  • wrapped bindings record the OUTBOUND MOCK-SET:    │ │
│              │   │    fetch · D1 · R2 · KV · DO · AI · Hyperdrive ·     │ │
│              │   │    Queue · Service-binding                           │ │
│              │   │  • ctx.waitUntil(sink.write(envelope))  ◀ off-path   │ │
│              │   └─────────────────────────────────────────────────────┘ │
│              │            │ envelope (≈4 KB)                               │
│   tail() ◀───┘            ▼                                                │
│   (metadata,     ┌────────────────┐                                       │
│    5xx trigger)  │ shadow_captures│──────────┐                            │
│                  │   R2 bucket    │          │  same corpus as Keploy     │
└──────────────────┴────────────────┴──────────┼────────────────────────────┘
                                               ▼
                              §17.4 workerd replay · §25 k6 load · catalog/anomaly

11.5 In-isolate middleware = the recorder

ShadowCaptureMiddleware wraps the same Service.toHono() boundary the visitor already attaches to (§11.1) — one wrapper, shipped to all 10 Workers because they share bindings (§7).

fun Service.toShadowHono(env: WorkerEnv, ctx: ExecutionContext): Hono {
  val base = this.toHono()                       // existing attach point (§11.1)
  if (!env.SHADOW_CAPTURE_ENABLED) return base

  return base.use { c, next ->
    if (!sampler.shouldCapture(c.req)) return@use next()

    val captureId = Uuid.v7().toString()
    val resources = mutableListOf<ShadowResourceRef>()
    val reqCopy   = c.req.raw.clone()            // handler still reads the original stream

    // Instrument bindings so every dependency call appends a mock to `resources`.
    c.env = instrumentBindings(env, captureId, resources)

    val startedAt = Clock.System.now()
    next()                                       // run the real handler
    val res = c.res.clone()                      // tee response without consuming it

    val envelope = ShadowEnvelope(
      schemaVersion = "reaktor.shadow.v1",
      captureId     = captureId,
      source        = Source(env = "prod", service = env.SERVICE_NAME, runtime = "cloudflare-worker"),
      route         = RouteRef(c.req.method, c.req.routePath),
      request       = redactor.redact(reqCopy),
      response      = redactor.redact(res),
      resources     = resources.toList(),        // ← the outbound mock-set
      timing        = Timing(durationMs = (Clock.System.now() - startedAt).inWholeMilliseconds),
      startedAt     = startedAt,
    )
    // Flush OFF the hot path: the user's response returns first, capture adds no latency.
    ctx.waitUntil(sink.write(envelope))          // → shadow_captures R2 bucket
  }
}

Inbound (the request/response half): clone the Request so the handler still gets its body stream; capture status + redacted body by tee-ing the Response. Assembly and the R2 write live inside ctx.waitUntil, so capture is off the response path — same under-1%-p95 budget as the JVM visitor (§32).

Outbound (the mocks half): every binding the isolate touches is wrapped so each call appends a ShadowResourceRef and records the response as a replay mock. This is exactly Keploy's outbound-mock recording, done per-binding instead of per-socket:

// One wrapped binding (mirrors ShadowR2Binding in §13.1). Wrap-on-capture, replay-on-test.
class CapturingD1(private val inner: D1Database, private val ctx: CaptureContext) : D1Database {
  override suspend fun query(sql: String, params: List<Any?>): D1Result {
    val rows = inner.query(sql, params)
    ctx.resources += ShadowResourceRef.d1(
      templateHash = sha256(normalize(sql)),
      params       = ctx.redactor.redactParams(params),
      rowsRef      = ctx.sink.stash(rows),       // mock the replay stub will return
    )
    return rows
  }
}

Tail Worker — a Cloudflare Tail consumer (tail() handler) attached to each producer receives execution traces out-of-band, at zero added latency, but never request/response bodies or binding payloads. So it is not a recorder; it is the catalog + anomaly trigger: it writes the envelope index row, detects 5xx / uncaught exceptions, and flips the sampler to 100% for the offending route or debug.session_id cohort (feeding §20's auto-pin). Bodies always come from the middleware; the Tail Worker is the cheap always-on signal that decides what to capture in full.

// shadowTail — bound as the tail consumer of every producer Worker.
export default {
  async tail(events: TraceItem[], env: Env) {
    for (const e of events) {
      await env.CATALOG.index(toCatalogRow(e));            // queryable by time/route/community
      const failed = e.outcome !== "ok" ||
        e.logs.some(l => l.level === "error") || e.exceptions.length > 0;
      if (failed) await env.SAMPLER.escalate(e.scriptName, e.event, "100%-on-error");  // §11.3
    }
  },
} satisfies ExportedHandler<Env>;

Front proxy (fallback only) — for a Worker we cannot modify, a separate Worker on the route tees request/response bodies. It captures the HTTP surface only — it is blind to the downstream isolate's bindings, so it yields a Keploy-HTTP-shaped capture with no mock-set. Use it only where middleware can't be shipped. §11.6 goes lower still — the complete isolate capture surface.

11.6 Going lower: the complete edge capture surface

§11.5 captures the HTTP request/response and the named-binding mock-set — Keploy parity. The isolate exposes far more, and one runtime property makes total capture tractable instead of a bottomless pit.

The determinism boundary — why "everything" is finite

A Worker is single-threaded (one event loop per request, no shared-memory threads; cross-isolate messaging is structured-clone). And to mitigate Spectre, Cloudflare freezes the clock: Date.now() and performance.now() only advance after I/O — they return the time of the last I/O and do not move while your code runs (CF docs).

That yields a clean theorem. The complete set of inputs that can change an execution's outcome is:

{ the inbound event } ∪ { every I/O result } ∪ { every explicit entropy/clock read }

There is no hidden mid-computation timing to miss, because the runtime refuses to let code observe it. Capture that oracle and replay is bit-deterministic.

The five capture layers

Layer 0 · the entropy/clock oracle — record values, not seeds. Per-request shims, scoped by AsyncLocalStorage (the nodejs_als flag) so concurrent requests in one isolate never cross-contaminate, record the actual outputs of Date.now, performance.now, Math.random, crypto.randomUUID, crypto.getRandomValues, and setTimeout/setInterval delays. On replay the same shims return the recorded sequence — stronger than reseeding a PRNG (it also pins crypto randomness and the I/O-anchored clock).

// nodejs_als: one CaptureContext per request, propagated through every async hop.
const als = new AsyncLocalStorage<CaptureContext>();
const realNow = Date.now, realRand = Math.random, realUUID = crypto.randomUUID.bind(crypto);

// Record-the-oracle shims — no-op unless a capture is active; mirror-image on replay.
Date.now          = () => als.getStore()?.clock(realNow)   ?? realNow();
Math.random       = () => als.getStore()?.entropy(realRand) ?? realRand();
crypto.randomUUID = () => als.getStore()?.entropy(realUUID) ?? realUUID();
// performance.now, crypto.getRandomValues, timer delays: same pattern.

export default {
  fetch(req, env, ctx) {
    const cap = newCapture(req);                          // captureId, req.cf, cf-ray
    return als.run(cap, () => handler(req, instrumentEnv(env, cap), ctx));
  },
};

Layer 1 · every entrypoint, not just fetch. Wrap the whole module so all invocation types are captured, each with its own replay driver:

Layer 2 · a generic env Proxy — capture every binding, even ones we never enumerated and ones added later. Instead of per-binding wrappers, wrap env in a recursive Proxy: every property access and method call on any binding is intercepted generically, and every config-var read is recorded as the execution's dependency set (secret values redacted; the fact of a Secrets Store .get() is kept).

function instrumentEnv(env: Env, cap: CaptureContext): Env {
  return new Proxy(env, {
    get(target, prop: string) {
      const v = (target as any)[prop];
      if (typeof v === "function") return recordCall(prop, v.bind(target), cap); // method → mock
      if (isBinding(v))            return instrumentEnv(v, cap);                  // nested (DO stub, …)
      cap.recordVarRead(prop);                                                   // config dependency set
      return v;                                                                  // (secret values redacted)
    },
  });
}

Operation-level granularity, per binding:

Layer 3 · request/response internals + console + errors + streams.

• request.cf — colo, country / region / city, lat/long, timezone, asn / asOrganization, httpProtocol, tlsVersion, tlsCipher, clientTcpRtt, requestPriority, and bot-management / client-trust scores (where enabled). Normally invisible to app code; capturing it lets replay reconstruct the geo / TLS / bot context the handler actually saw.
• cf-ray — the edge request id. The join key that ties this in-isolate capture to its Tail event and its Logpush log (Layer 4).
• Full headers (allowlisted/redacted), the request AbortSignal (did the client disconnect?), HTTP version/method/URL.
• Response status + headers (incl. cf-cache-status), streaming/SSE chunk boundaries + per-chunk timing, 101 WebSocket upgrades, caches.default hits.
• console.{log,warn,error,debug} arguments + level (ALS-scoped, so logs attribute to the right capture under concurrency).
• Thrown exceptions + normalised stack (top frames), unhandled rejections, passThroughOnException, and limit-exceeded (CPU / memory) errors.

Layer 4 · out-of-band truth — zero added latency, complements the in-isolate planes.

• Tail Worker TraceItem: cpuTime, wallTime, outcome, logs, exceptions, event, diagnostics-channel events. This is the only place CPU/wall time exists — the isolate cannot self-measure it (the clock-clamp forbids it), so the perf oracle must come from here (CF docs).
• Logpush → R2: the canonical edge HTTP log (edge status, WAF actions, bot score, cache, colo, TLS) joined by cf-ray. The audit / ground-truth plane.

                   cf-ray  ──  the join key across all three planes  ──┐
                                                                       │
 IN-ISOLATE  (middleware + env Proxy + ALS oracle)                     │
   bodies · binding mock-set · entropy/clock oracle · request.cf       │  replay-grade
   console · exceptions · stream chunking          ──────────────┐     │  (bit-exact)
                                                                 ├──▶ R2 corpus ◀──┐
 TAIL WORKER  (out-of-band, ~0 latency)                          │                │
   cpuTime · wallTime · outcome · log/exception summary  ────────┤  ONLY source   │
                                                                 │  of CPU/wall    │
 LOGPUSH → R2  (edge audit)                                      │                │
   WAF · bot score · cache · colo · TLS · edge status   ─────────┘  ground truth ─┘

The honest ceiling — and why it costs profiling, never replay

Cost tiers — full capture isn't free

Tiering keeps the under-1% p95 budget (§32) on normal traffic while making any flagged request — an error, a debugged user, a pinned incident — deterministically replayable down to the entropy stream.

12. Shadow envelope

The envelope is the unit of capture. Compact (~4 KB compressed FlexBuffer), self-describing, replayable.

{
  "schemaVersion": "reaktor.shadow.v1",
  "captureId":     "01HXKQF8M7XYZ...",        // UUID v7 (time-sortable)
  "traceId":       "00-a3c.../b2d.../...",
  "capturedAt":    "2026-05-28T10:30:00Z",
  "source":        { "env": "prod", "service": "messaging-service", "runtime": "cloudflare-worker" },
  "tenantId":      "shadow_bestbuds",
  "route":         { "method": "POST", "pattern": "/messages/{chatId}", "transport": "HTTP" },
  "graph":         { "appId": "bestbuds", "nodeId": "ChatInteractor", "community": "chat" },
  "request": {
    "headersAllowlist": { "traceparent": "...", "x-app-build": "..." },
    "bodyRef":          "r2://shadow-raw/2026/05/28/body.req.flex",
    "redaction":        "policy:user-content-v1"
  },
  "response": {
    "status":     200,
    "bodyRef":    "r2://shadow-raw/2026/05/28/body.res.flex",
    "comparator": "shape-and-stable-fields"
  },
  "resources": [
    { "type": "supabase",       "table": "messages", "keys": ["msg_..."], "mode": "read-write" },
    { "type": "durable-object", "class": "ChatRoom", "idHash": "room_...", "snapshotRef": "r2://..." },
    { "type": "r2",             "bucket": "IMAGES",    "keyHash": "img_...", "transform": "placeholder" },
    { "type": "memgraph",       "labels": ["User", "Room"], "subgraphRef": "r2://..." },
    { "type": "ai",             "model": "@cf/meta/llama-3", "promptHash": "sha256:...", "durationMs": 340 }
  ],
  "timing":         { "durationMs": 84, "dbMs": 22, "workerCpuMs": 8 },
  "classification": { "sample": "rolling", "anomalyScore": 0.12, "retention": "7d" },
  "pinned":         false,
  "capsuleClass":   null                          // set if pinned: "rolling"|"latency"|"crash"|"anomaly"|"incident"
}

13. Data plane & materializers

The shadow environment needs data closure, not a blind production clone. A captured trace is useful only if the rows, objects, graph nodes, DO state, and feature flags it depends on can be materialised together.

13.1 Read-through R2 binding (illustrative)

class ShadowR2Binding(
  private val shadow: R2Bucket,        // shadow_IMAGES
  private val production: R2Bucket,    // IMAGES (read-only)
  private val transformer: MediaTransformer,
) : R2Bucket {
  override suspend fun get(key: String): R2Object? {
    shadow.get(key)?.let { return it }
    val upstream = production.get(key) ?: return null
    val placeholder = transformer.transformIfSensitive(upstream)
    shadow.put(key, placeholder.body, metadata = mapOf("shadow.cached" to "true"))
    return placeholder
  }
  override suspend fun put(key: String, body: ByteArray) = shadow.put(key, body)
}

14. Shadow capsules

A shadow capsule is the unit of replay. It packages traffic, resource closure, expected outcomes, comparators, and retention metadata. Rolling capsules expire quickly. Pinned capsules preserve incidents and weird edge cases.

14.1 Capsule classes

14.2 Pinning rules

• Pin any trace with 5xx, timeout, retry exhaustion, DLQ write, crash breadcrumb, auth rejection after valid input, media upload failure, or data-integrity violation.
• Pin any trace whose latency is above route budget by 2× or in the top 0.5% for a service.
• Pin any trace manually linked to a support ticket, incident, customer complaint, or debugging session.
• Deduplicate by route + error class + stack fingerprint + graph community + resource shape hash.
• Retire only after a replay test exists, passes on main for 14 days, and the owning team accepts removal.

14.3 Pin budget

Anomaly detector pins at most 50 captures per day per class. Excess go into a candidate queue for human curation in the Workbench. Prevents the pinned set from ballooning to where nobody knows what's in it.

15. The 7-day window + pinned fixtures

ROLLING WINDOW                         PINNED CAPSULES
──────────────────────                 ─────────────────────
moves with wall clock                  permanent (per class TTL)
~7 days deep                           5,000 → 50,000 entries
~50 GB at peak                         grows slowly
auto-trimmed nightly                   human + AI curated

┌─────────────────────────┐            ┌──────────────────────────┐
│ captures from last 7d   │ ─ promote ▶│ pinned (5 classes)       │
│ - 5% sample of normal   │ on anomaly │ - the OOM from January   │
│ - 100% on error/slow    │ / manual   │ - 12k-friend user repro  │
│ - 100% for debug.uid    │ pin        │ - chat-init race         │
└─────────────────────────┘            └──────────────────────────┘
            │                                      │
            ▼                                      ▼
     trim job: 0 4 * * *          retention per capsule class
                                  (Crash 180d, Latency 30d, etc.)

15.1 Storage maths

Average envelope ~4 KB (compressed FlexBuffer). At 50 rps production × 5% sample = 2.5 captures/sec = 216,000/day = 6.5M/week. At 4 KB each that's ~26 GB for the rolling window. Pinned set assumed to grow to ~50,000 entries over 12 months = ~200 MB. Postgres shadow data at 7-day cut of BestBuds today is ~12 GB. Total shadow R2 + Postgres footprint at maturity: under 50 GB — fits in a single Hyperdrive-backed Postgres + R2 free tier.

16. Materialization workflow

Materialisation converts capture refs into a runnable shadow environment. Deterministic, repeatable, cheap enough to run nightly.

Step 1 · catalog query

Select window + capsules

Representative rolling traffic from last 7 days, then union pinned capsules relevant to target env. Partition by graph community so small PRs don't hydrate the world.

Step 2 · dependency graph

Compute resource closure

Walk resource refs for every selected trace: Supabase rows, R2 objects, Memgraph nodes/edges, feature flags, provider stubs. Add parent rows and FK dependencies.

Step 3 · policy engine

Redact & tokenize

Deterministic tokenisation for ids and relationship fields, synthetic replacements for direct PII, media transformations, geo bucketing, header allowlists. Emit redaction report with rejected fields.

Step 4 · provisioner

Hydrate targets

Create / reset shadow Supabase schema, R2 prefixes, Memgraph instance, Workers env bindings, non-prod secrets. Base is immutable; overlays take writes.

Step 5 · quality gate

Verify replay

Replay a smoke subset and verify status, schema shape, stable fields, graph side effects, idempotency records, resource writes. Publish a ShadowManifest artifact.

17. Replay engine

Replay has two layers. Keploy-style HTTP replay verifies externally-visible API behaviour. Reaktor replay verifies the full service graph: DO state transitions, queue messages, Pub/Sub events, R2 writes, Memgraph mutations, repository side effects.

17.1 Runners

17.2 What gets replayed

Replay isn't just "re-execute the request" — it reconstructs the captured causal chain:

1. Set Clock, Random, Uuid to deterministic streams seeded from captureId.
2. Wrap every external binding (R2, Workers AI, Sheets, FCM) so calls matching the captured resources return the captured response.
3. Issue the captured request with all original headers preserved (except x-trace, which gets a fresh one tagged shadow.replay-of=<captureId>).
4. Compare the new response + downstream chain to the captured baseline per the comparison policy (next section).
5. Report diffs structurally: status mismatch, header diff, body diff (FlexBuffer-aware path-level), missing downstream call, extra downstream call, latency regression.

17.3 Current Keploy/R2 adapter

The current adapter is a four-step bridge:

1. npm run keploy:record:start talks to Keploy k8s-proxy and starts recording namespace=reaktor, deployment=app-server, with static deduplication and health-check filters.
2. npm run keploy:record:stop downloads the active Keploy test bundle before stopping the session, uploads bundle.zip and metadata.json to s3://$R2_BUCKET/keploy/reaktor-server/<date>/<record-id>/, then stops the proxy session.
3. npm run keploy:r2:pull downloads the latest or named R2 bundle into targets/reaktorServer/.keploy/current.
4. npm run keploy:test runs keploy test -p <current> using targets/reaktorServer/keploy.yml and a configurable KEPLOY_COMMAND.

This keeps capture in k3s and replay local, which is the intended capture/replay split: the cluster observes real API traffic, while the developer machine runs the regression suite without needing to reproduce cluster traffic generation.

17.4 Edge replay in workerd (the Keploy-equivalent run)

Keploy replays by booting the JVM app and feeding recorded inbound requests while stubbing the recorded outbound mocks. The edge equivalent boots the Worker in workerd — Cloudflare's open-source runtime, the same engine wrangler dev and Miniflare run — so replay happens locally and in CI without touching the edge:

1. Load the Worker bundle into workerd via Miniflare / unstable_dev (programmatic, headless).
2. Replay the entropy/clock oracle (§11.6, Layer 0): the same ALS-scoped shims that recorded Date.now / performance.now / Math.random / crypto.random* now return the captured values in order — determinism by recording, not by reseeding. The §32 BlockHound-equivalent is an AST / ESLint rule that bans raw use of these outside the shim.
3. Install stub bindings through the same generic env Proxy (§11.6, Layer 2), in reverse: D1 returns the recorded rows, R2 the recorded object / placeholder, AI the CachedByPrompt completion, fetch the recorded subrequest response, Queue / Service-binding the recorded intent. Non-idempotent calls hit the §19 StubRegistry — never the network.
4. Drive the right entrypoint (§11.6, Layer 1): a fetch replays the request; a queue batch, scheduled tick, email, DO alarm, WebSocket message sequence, or RPC call is re-driven from the capture. Workflow steps replay natively from their memoized step cache.
5. Compare response + downstream mock-set per the envelope's comparator (§18); CPU/wall deltas come from the Tail plane (§11.6, Layer 4), since the isolate cannot self-measure them.
6. Emit the same structured diff as Keploy: status, body (FlexBuffer path-level), missing / extra subrequest or binding call, entropy divergence, latency regression.

shadow replay capsule <id>          # dispatches by tier, automatically:
#   JVM / k3s capture  → Keploy CLI (HTTP) or Reaktor service replay
#   Workers capture    → workerd harness (inbound request + binding mock-set)

Both runners read the same R2 corpus and emit the same diff format, so the catalog, comparators (§18), CI gates (§29), and Workbench panels (§28) are tier-agnostic.

17.5 The complete testing stack

Capture/replay is one layer of a six-tool stack. Each tool owns a tier; all of them read or feed the one R2 corpus, so a single captured production interaction can become a regression test, a contract assertion, a load profile, or a UI journey without re-recording.

  CAPTURE — 2 engines, split by runtime              CONSUME — 1 corpus, many runners
  ───────────────────────────────────────            ──────────────────────────────────
  Keploy (k8s-proxy)  ──▶ k3s / JVM tier ──┐
  Shadow Worker Proxy ──▶ Cloudflare edge ─┤
   (middleware + Tail Worker)              ├──▶ ┌────────────────────────┐
                                           │    │  R2 corpus             │
  Tail Worker = catalog + anomaly trigger ─┘    │  shadow_captures/  +   │
                                                │  keploy/               │
                                                └───────────┬────────────┘
              ┌──────────────┬──────────────┬───────────────┼───────────────┐
              ▼              ▼              ▼               ▼               ▼
        workerd replay   Keploy CLI      Karate          k6 (§25)     Playwright /
          (edge)        (JVM HTTP)   (API contract)    (load 1–5×)    Maestro (UI)

17.6 The replay frontier — how far this goes

§11.6 proved the capture floor: in production the Spectre clock-clamp hides everything below the I/O boundary. Replay inverts that. A deterministic capsule is portable, and workerd running locally does not clamp the clock (timers advance in local dev). So once a run is reconstructed, the floor lifts — you can observe, perturb, compare, and recombine the execution without limit. Each frontier below is something the deterministic capture unlocks that raw production never could.

1 · Omniscient replay — debug below the I/O floor

Because replay is deterministic and local, attach the V8 inspector over the Chrome DevTools Protocol (wrangler dev --inspect, or workerd's --inspector-addr): set breakpoints, step, take heap snapshots, and CPU-profile the exact production input — none of which the edge isolate permits in production. This is the Replay.io / rr "record once, debug forever" promise, achieved at the service layer. Precision caveat: workerd intercepts the Profiler domain's precise-coverage path (workerd#2065), so line/branch coverage uses Istanbul source instrumentation (the path the Workers Vitest pool takes), while breakpoints, heap snapshots, and Wrangler CPU profiles work directly.

2 · Differential replay — version × version, local × edge, runtime × runtime

One capsule, two targets, a structural diff:

• version × version — replay against deployed version N vs N+1 by setting the Cloudflare-Workers-Version-Overrides header (or a versioned preview URL <ver>-<worker>.<subdomain>.workers.dev). Semantic regression detection before a gradual rollout reaches real users.
• local × edge — workerd vs wrangler dev --remote. Catches the exact "worked locally, broke on the edge" class this whole doc opens on.
• runtime × runtime — the same Reaktor Service exists as a JVM handler (Keploy-captured, k3s) and a Worker (Shadow-captured, edge). Both emit the same envelope schema, so a JVM capture can be replayed against the Worker implementation and vice-versa — proving behavioural equivalence across runtimes. The Reaktor multiplatform promise, tested rather than asserted.

3 · Live shadow execution — diff before you promote

Beyond capture-then-replay-later: for a sampled request, the production worker also invokes the candidate version in-isolate (service binding / version override held at 0% traffic) with the same inputs, every side-effecting binding stubbed, and diffs the two outputs inside ctx.waitUntil — discarding the shadow result. Dark-traffic / shadow-deploy (GoReplay's idea, §5) realised in the isolate, gated on the §9 side-effect classification so the shadow can never emit a real effect. Divergences feed the corpus and block promotion.

4 · Beyond regression — fuzzing & fault injection from real captures

• Coverage-guided fuzzing. The corpus is a seed corpus: mutate the captured inbound request and the recorded mock-set (Layer 2) and re-run hermetically in workerd, Istanbul-coverage-guided, to discover unhit paths and crashes — AFL-style, seeded by real production shapes instead of synthetic noise.
• Fault / chaos replay. Replay the capture but flip recorded mocks: D1 success → error, R2 hit → miss, injected latency, a truncated stream, a forced 1102 CPU-exceeded or subrequest-limit hit. Real inputs, adversarial dependencies — does the handler degrade gracefully?
• Metamorphic invariants. Replay twice → identical side-effects (idempotency); reorder independent subrequests → identical result (commutativity). Catches bugs an exact-diff can't even express.

5 · Distributed super-capture — replay the transaction, not one hop

A single user action fans out: Worker A → service binding → Worker B → Durable Object → Queue → consumer C → JVM server. Each hop is captured independently; stitch them by traceparent + cf-ray into one causal-DAG capsule and replay the entire distributed transaction in topological order across a combined workerd + JVM harness. The graph kernel makes the DAG first-class — this is §17.2's "replay the causal chain" taken to its distributed, multi-runtime limit: one capsule = one whole transaction.

6 · Stateful replay — Durable Objects to the bit

Durable Objects are the hard case, but their only nondeterminism is the gate schedule: input gates block new events during synchronous execution, an await opens the gate to let requests interleave, output gates hold writes until storage commits, and blockConcurrencyWhile serialises (CF: rules of DOs). Capturing the gate open/close ordering + storage ops + the WebSocket/alarm timeline = capturing the DO's entire "thread schedule." Restore its SQLite to the exact pre-request state via PITR bookmarks (getCurrentBookmark / onNextSessionRestoreBookmark, 30-day, §11.6), then re-drive the captured timeline → deterministic stateful replay, hibernated WebSocket sessions included.

7 · The self-curating corpus — production is the test suite

Replay is hermetic and cheap, so shard it and replay a representative cover set against every PR. Compute that set by coverage-minimal corpus minimization — greedy set-cover over the captured Istanbul coverage profiles, yielding the smallest capsule set that still exercises every production path (afl-cmin for live traffic), scoped by graph community (§4). The corpus continuously curates itself toward "the minimal set of real requests that covers all of production."

8 · Capture without seeing — the privacy frontier

The forensic tier (Layers 0 + 3) is PII-dense, so push privacy as far as the capability allows: envelope bodies encrypted at the edge under a key only the replay enclave holds (the R2 corpus is ciphertext at rest); replay inside a confidential-compute context; differential-privacy noise on the §25 load aggregates. The deterministic-faker key (§21.2) already lets devs join across tables without real identities — encryption closes the at-rest gap so "capture everything" never means "anyone can read everything."

# the frontier verbs
shadow replay <id> --inspect              # attach CDP: breakpoints, heap, CPU profile
shadow diff-versions <id> --a v1.4 --b v1.5    # version × version structural diff
shadow shadow-run --sample 1% --candidate next # live dual-run, side effects stubbed
shadow fuzz <id> --mutate request,mocks --cover # coverage-guided fuzz from a capsule
shadow replay <id> --as-dag               # stitch + replay the whole causal transaction
shadow cover --minimize --community chat  # compute the coverage-minimal capsule set

18. Response comparison policy

Not all responses should be compared the same way. The capsule declares the policy at capture time:

The replay engine picks the policy from the envelope's response.comparator field. Three replay modes wrap this further: strict-mode CI gates on any diff under the policy; tolerant-mode allows policy-tolerated diffs but flags new diffs; exploratory reports everything without failing (interactive debug).

19. Stubbing dangerous side effects

The replay engine consults a StubRegistry before any external call leaves the shadow:

object StubRegistry {
  private val stubs = mapOf<CallKind, StubStrategy>(
    CallKind.AI_TEXT_GENERATION  to CachedByPrompt,
    CallKind.SHEETS_APPEND        to NullSink(captureFor = "sheets-writes.jsonl"),
    CallKind.FCM_SEND             to NullSink(captureFor = "fcm-sends.jsonl"),
    CallKind.APNS_SEND            to NullSink(captureFor = "apns-sends.jsonl"),
    CallKind.SUPABASE_AUTH_LINK   to DeterministicSuccess,
    CallKind.WEBHOOK_OUTBOUND     to NullSink(captureFor = "webhooks-out.jsonl"),
  )

  fun strategyFor(call: CallKind): StubStrategy =
    stubs[call] ?: if (call.isIdempotent) PassThrough else FailWithGuidance(call)
}

FailWithGuidance throws an exception with a message like "Call to POST https://api.stripe.com/charges blocked in shadow mode. Add a stub via StubRegistry.register(CallKind.STRIPE_CHARGE, …) or replay with --allow-real-effects." — surfaces the dangerous call instead of silently letting it through.

20. Anomaly detection & auto-pinning

A scheduled Worker runs every 15 min over the rolling capture stream and pins anything unusual:

• Stack-trace clustering — captured 5xx errors with stack traces hashed (top 10 frames, normalised). A new cluster (or one not seen in 30 days) auto-pins one exemplar.
• Latency outliers — per-endpoint p99.9 over 24h; any capture exceeding it pins with reason slow_outlier.
• Payload extremes — top-1% input size (largest payload, deepest graph, most friends) pins.
• Schema-edge detection — track which combinations of optional fields appear in request bodies. A combination not seen in last 30 days auto-pins.
• Causal-chain depth — captures whose resources.length is above p99 (deep graph fan-out) pin.
• Dedup by fingerprint (§14.2). Repeat occurrences increment a counter instead of pinning new capsules.

21. Privacy & redaction contract

21.1 Per-data-class handling

21.2 Deterministic faker key

A per-tenant secret HMAC key, scoped only to the capture-time scrubber. Same key for the life of the shadow cluster — so the same user is always "Alice Wonderland" in shadow, every replay. Devs can join across tables without seeing real names.

21.3 Auditing

Every capture's scrubManifest records which fields were scrubbed and by which rule. A nightly compliance job samples 100 random captures, decrypts the manifest, and verifies no raw PII slipped through (e.g. a free-form text field that wasn't annotated). Failures alert the data-team channel.

22. Access model

• Developers — read access to shadow data; write access only to their overlays.
• Production capture service — write-only access to raw shadow lake; no read access to developer overlays.
• Materialisers — service principals with explicit scopes: Supabase export, R2 copy, DO snapshot, Memgraph export. No cross-scope.
• Audit — every capsule read, hydrate, replay, and export emits an audit record (who, when, capsuleId, environment).
• Secrets — all shadow secrets live in GCP Secret Manager and sync to Cloudflare Worker shadow environments as non-prod secrets.

23. Tenant isolation

Three layers of "you can't accidentally hit prod":

1. ID prefix — all shadow tenant IDs, user IDs, room IDs are rewritten shadow_. Prod has no such IDs; any query with the prefix returns empty against prod, any query without returns empty against shadow.
2. Bucket / database prefix — separate R2 bucket (shadow_IMAGES), separate Postgres database (shadow_db), separate Memgraph (shadow-memgraph on a sandbox GCP project).
3. Credential isolation — shadow credentials are scoped at the Cloudflare account + GCP project level. The prod service account literally cannot reach shadow infrastructure and vice versa.

24. Time travel

Every capture envelope carries capturedAt. Replay can pin to any historical moment within the retention window:

shadow replay --as-of 2026-05-20T14:30:00Z --endpoint /api/feed --since 1h

The engine restores shadow Postgres + Memgraph to that timestamp via SELECT * FROM ... AT TIMESTAMP (CockroachDB native), or Postgres logical replay rewind, then runs the requested captures. For points outside the 7-day window, the cold restore from bestbudsArchiver's nightly backup is used (slower, ~10 minutes to materialise).

25. Load testing from real patterns

The capture corpus is also a load-test corpus. Per spec §28.2:

shadow load --since 7d --speedup 10x --target http://staging-shadow --duration 30m

# Internally:
# 1. Sample captures from last 7d, weighted by endpoint frequency in prod.
# 2. Replay them at 10× the original arrival cadence.
# 3. k6 collects per-endpoint latency histograms, errors, throughput.
# 4. Export to Mimir; Grafana panels render against the performance-observability
#    pipeline (see /docs/performance-observability).

Pattern fidelity: traffic is replayed with the same inter-arrival distribution as production — bursts, idle periods, diurnal cycles — not as a uniform RPS stream. This catches issues that uniform load tests miss (queue depth spikes, cache stampedes).

26. Observability

Every capture and replay emits OTel spans into the same Grafana LGTM pipeline described in performance-observability §17. Two service names:

• service.name = "shadow-capture" — the visitor itself. Tells you capture overhead, sample drops, scrubber failures.
• service.name = "shadow-replay" — every replay run. Tells you replay duration, diffs detected, false-positive rate.

The Workbench gets a Shadow tab showing: captures/sec live, pinned set size, oldest unpinned capture, today's replay diff count, anomaly auto-pin rate, replication lag.

27. CLI verbs

# Pull the current moving shadow into local config
shadow env pull rolling-7d

# Boot BestBuds with shadow bindings and disposable overlay
shadow run local -- ./targets/reaktorServer/dev.sh

# Replay one incident capsule against a local branch
shadow replay capsule incident-chat-send-timeout-2026-05-21

# Run route-mix load test generated from captured traffic
shadow load rolling-7d --rate 2x --duration 15m

# Refresh Cloudflare Worker secrets/bindings for shadow env
shadow sync cloudflare --env shadow-current

28. Workbench integration

The Reaktor workbench should make the shadow environment visible and usable. Not just dashboards — let developers select a graph community, inspect captured traces, hydrate a shadow dataset, replay a capsule, and see the exact graph nodes and resources involved.

29. CI integration

Three CI jobs per PR, all running against the shadow stack:

CI · job 1~2 min

Replay pinned regression suite

shadow replay --class crash,incident --strict --against $PR_BRANCH. Every pinned crash and incident capsule must still produce the captured response. Failure blocks merge.

CI · job 2~3 min

Replay yesterday's traffic sample

shadow replay --since 24h --sample 1% --tolerant --community $CHANGED_COMMUNITY. A random 1% of the last day's captures replayed in tolerant mode, scoped to graph communities the PR touched. Catches drift that pinned fixtures don't cover.

CI · job 3~10 min (nightly)

Load test from last week's patterns

shadow load --since 7d --speedup 5x --duration 8m. Nightly on main, not per-PR. Latency histograms feed the perf dashboard; regressions vs previous nightly create a ticket.

30. BestBuds wiring

30.1 First slice: chatServer

Start with one worker — chatServer — because it has the richest traffic and an obvious high-value debug surface. Validate the capture path on one worker before fanning out to the other nine.

31. Phased plan

Current slice 2026-06-04

Keploy + R2 HTTP adapter

Keploy k8s-proxy install script, k3s record/status/stop scripts, R2 archive script, local R2 pull script, local Keploy replay script, and root npm commands exist. DoD: app-server HTTP traffic can be captured in k3s, archived to R2, pulled locally, and replayed with npm run keploy:test.

Phase 0~1 week

Policy & schemas

Define ShadowEnvelope, ShadowResourceRef, ShadowCapsule, redaction policies, retention classes, access roles, comparator types. FlexBuffer schemas + serde + unit tests on round-trip.

Phase 1~1-2 weeks

Service capture

ShadowCaptureVisitor + ShadowServiceInterceptor on chatServer first. Captures envelopes to shadow_captures R2 bucket. PII scrubber for chat-message fields. 5% sampling. On the edge tier this is ShadowCaptureMiddleware + the chatServer Tail Worker, with fetch / D1 / R2 / AI binding instrumentation recording the mock-set (§11.4–§11.5); on the JVM tier it is the visitor + the existing Keploy adapter. DoD: 24h of chatServer traffic (edge + JVM) visible as well-formed envelopes, <1% capture overhead per request.

Phase 2~1 week

Shadow lake + catalog

Raw sanitised bodies → R2. Envelope index in D1 / Postgres. Catalog queries by time, route, service, graph community, anomaly fingerprint. DoD: shadow inspect + shadow catalog query work end-to-end.

Phase 3~2-3 weeks

Data materializers

Supabase row closure via logical replication. Memgraph subgraph export / import. R2 read-through. Stubs for Workers AI, Sheets, FCM. DoD: a dev can run chatServer locally against full shadow stack and produce no real side effects.

Phase 4~2 weeks

Replay runners

Reaktor replay runner, promotion of the existing Keploy adapter into shadow replay, the workerd edge-replay harness (the Keploy-equivalent for Workers, §17.4), k6 generator, response comparators, side-effect checkers, report artifacts, and wiring the existing target-owned Karate gate to assert against replayed captures. DoD: a captured request from yesterday — JVM or Worker — can be replayed against a local branch and produces a structured diff across the full service graph (incl. the binding mock-set), not only HTTP.

Phase 5~½ week

Anomaly detector + pin queue

15-min scheduled Worker: stack-trace clustering, latency outliers, payload extremes. Pin budget (50/class/day). Candidate queue surfaces in Workbench.

Phase 6~1 week

Roll capture to all 10 workers

Apply Phase-1 middleware to every worker. Tune per-endpoint sample rates. Validate scrubber coverage on every endpoint's schema.

Phase 7~2 weeks

Dev + CI integration

shadow CLI, PR overlays, GitHub Actions workflow, Workbench panels, release gates. DoD: a +500 KB regression PR fails CI; +100 ms cold-start regression posts a Warning comment.

Phase 8~1 week

Load test + time travel + per-branch forks

shadow load verb + k6 integration. shadow restore --as-of. Supabase branching for per-PR DB forks (if account tier supports it).

32. Risks & open questions

Risks

• PII leaks into dev. Fail-closed redaction policy, field classifiers, denylisted headers, automated scans before materialisation, audit logs for capsule access. A new free-form text field added without @Pii annotation could leak real PII — nightly sampler catches this.
• Shadow accidentally writes to production. Separate credentials, Cloudflare envs, Supabase / R2 / Memgraph targets, network-egress guardrails, write-block tests. The shadow_ ID prefix is the last-line defence.
• Durable Object state not enumerable. When DOs come in, add explicit snapshot endpoints + mailbox logging per DO class; replay from mailbox when snapshots are unavailable.
• Dataset becomes too expensive. Rolling 7-day default, sampled success traffic, pinned anomaly dedupe, R2 lifecycle rules, materialise by graph community.
• Replay non-determinism. Freeze clocks, stub third-party providers, compare allowed drift, seed ids, preserve idempotency keys, record dependency responses. BlockHound-style runtime check in CI catches System.currentTimeMillis, UUID.randomUUID, etc. outside framework facilities.
• Schema drift breaks old capsules. Version every envelope and capsule, run migration adapters, mark un-migratable capsules as archived but inspectable.
• Developers distrust the environment. Publish freshness, source traffic window, replay pass rate, redaction report, drift score in the Workbench. The single biggest adoption risk: a dev who once sees a "shadow says X but prod did Y" inconsistency stops trusting shadow forever.
• Capture overhead in prod. Target <1% latency overhead at p95 with 5% sampling. Mitigation: capture envelope assembly async, writes batched to R2 every 200 ms, dropped on backpressure.
• Logical replication lag. If shadow DB lags prod by minutes, recent captures replay against stale data. Monitor as first-class metric; alert if > 30 s.
• Stub registry stales. Prod adds a new external API; stub registry doesn't know. FailWithGuidance default + CI scan for new HTTP-client construction.

Open decisions

• Catalog index — D1, Postgres, or both?
• How much free-text chat content can be safely preserved after tokenisation?
• Every PR gets a full overlay or only graph-community overlays?
• Memgraph shadow on existing GCP CE host, second CE host, or local containers for developer mode?
• Should the existing Keploy npm scripts stay as an ops tool, or should shadow replay wrap them so developers never call Keploy directly?
• Shadow stack in a separate Cloudflare account vs same-account-different-zone? Separate is safer; same is cheaper.
• Does the bestbudsArchiver nightly backup become the cold-restore source, or a separate path?
• How does this compose with reaktor-notifications' outbox when that lands? Outbox rows are perfect captures.
• Shadow access for external Reaktor contributors (if open-sourced)? Probably no — captures still encode tenant-specific shapes.

33. References

Spec sections this design extends

• §28 Reaktor Shadow: Testing & Quality Infrastructure — "capture as a visitor" foundation this doc operationalises into a permanent dev environment.
• §21.8 Transactional outbox — pattern that lets us safely capture publish intentions without firing them in shadow.
• §21.9 Idempotent consumers — replay correctness depends on this.
• §29 Performance: Measurement, Profiling, and Optimization — load testing & OTel pipeline this doc plugs into.
• §29.4 Automated performance testing — Keploy / k6 / Maestro integration assumed.

Peer designs in this repo

• performance-observability — Grafana LGTM pipeline shadow capture/replay flows into; Keploy + k6 driver runners.
• reaktor-notifications-design — uses outbox; FCM/APNs stubs from §19 reuse the same null-sink pattern.
• deep-link-integration — same graph-visitor pattern; a deep-link tap is a typical capture target.
• auth-analysis — service principals + tenant API keys; shadow runs as a dedicated service principal.
• security-plan — PII scrubbing rules; shadow extends them with the deterministic-faker key.
• graph-kernel-analysis — visitor + facets foundation.
• reaktor-graph-visitors-design — base classes ShadowCaptureVisitor extends.
• claude-dev-environment — prior detailed draft (1,087 lines); preserved for design lineage.
• codex-dev-environment — prior alignment draft (635 lines); source of the capsule taxonomy, materialisation workflow, environment-class table, and response comparison policy.

Files inventoried during the design

• bestbuds/wrangler.json (root) — Hyperdrive: supabase, supabasedirect, cockroach, cockroachdirect; R2: IMAGES.
• bestbuds/targets/{analytics,chat,sticker,config,bot,bestbudsArchiver,media,notification,social,messaging,prototype}{Server,Worker}/wrangler.json — 10 production workers + 1 archiver + 1 prototype.
• bestbuds/targets/reaktorServer/src/main/kotlin/ai/bestbuds/server/ServerApplication.kt — Spring Boot REST + GraphQL surface, memgraphDriver().
• bestbuds/targets/reaktorServer/.../MannaKnowledgeGraphGraphQl.kt — Cypher queries; labels (Domain, Resource, Chapter, Tenant, Task) and edges (ChapterEdge, TaskDependencyEdge, TaskResourceEdge, GoalTaskEdge).
• bestbuds/modules/app/src/jvmMain/kotlin/ai/bestbuds/app/db/schema.sql — Postgres schema.
• bestbuds/n8n/workflows/ — 10 deploy / build / sync workflows; none for shadow refresh yet.

External tools cited

• Keploy — capture/replay regression testing
• Keploy Kubernetes Proxy REST API
• Keploy Kubernetes setup
• Cloudflare R2 S3 API
• GoReplay — TCP traffic mirroring
• mirrord — local-process-in-remote-pod
• Replay.io — deterministic record/replay
• Supabase Branching
• Neon Branching
• Memgraph DUMP DATABASE
• Supabase logical replication
• Cloudflare R2 event notifications
• k6 load testing
• Cloudflare Tail Workers — trace events
• workerd — the open-source Workers runtime
• Miniflare / wrangler unstable_dev — local Worker replay
• Cloudflare Workers — performance & timers (Spectre clock-clamp)
• Cloudflare Workers — AsyncLocalStorage (nodejs_als)
• Cloudflare Workers Trace Events — cpuTime / wallTime fields
• Durable Objects — point-in-time recovery bookmarks
• Cloudflare Workers — breakpoint debugging via DevTools (CDP)
• Profiling your Workers with Wrangler — CPU profiles
• Cloudflare Workers — version overrides (Cloudflare-Workers-Version-Overrides)
• Durable Objects — input/output gates & rules
• workerd#2065 — V8 precise-coverage interception (use Istanbul)
• Karate — API test & contract automation
• karate-gatling — performance from Karate features
• Playwright — web E2E
• Maestro — mobile E2E

Doc lineage. Authored 2026-05-28. Merged from codex-dev-environment (12 sections — capsule taxonomy · materialisation workflow · environment classes · response comparison policy · redaction contract · access model) and claude-dev-environment (24 sections — visitor code · envelope JSON · side-effect tri-classification · storage maths · prior-art table · CLI verb catalog · 3 CI jobs · 9-phase plan). Both prior drafts preserved in /docs/ for design lineage. Grounded in Reaktor Spec §28 and a fresh inventory of BestBuds' production data surfaces.

Related docs

• Perf and Observability - Use before optimizing or adding instrumentation.
• DevOps - Use before changing the build/deploy/ops pipeline, the Kotlin platform model, deployment partitioning, gated deploy operations, or the reaktorDesktop Deploy pane; pairs with reaktor-cloud-pane (operator view) and secrets-management.
• Reaktor Secrets - Use before moving, rotating, distributing, caching, or materializing secrets.
• Security Plan - Use before enabling deploy actions, remote agents, or privileged commands.
• Claude Dev Env - Use only when comparing prior shadow environment proposals.
• Codex Dev Env - Use only when comparing prior shadow environment proposals.