Shadow Verification and Failure Capsules

Adapted from textbook chapters 12 and 14. Baseline equivalence is one kind of evidence. Production can contain a bug, so the candidate must also satisfy independently defined product and safety contracts.

Three checks

Version normalization and comparator policy. Remove only declared nondeterministic fields; do not normalize away ordering, authorization or duplicate effects that matter. Both executions use the same sanitized world and equality relationships. A production trace is contextual evidence unless its pre-state/cut supports a controlled comparison.

Reuse Kotlin tests, Karate, Playwright, Maestro, k6, Keploy and actual Worker-runtime tests for the layer they exercise. Schemathesis can explore API-schema behavior and ZAP can automate scoped security checks; product-level auth and business invariants remain explicit assertions. Schemathesis, ZAP Automation Framework.

Privacy and effect safety

Classify capture fields and artifacts before export. Preserve the equality, relationships, cardinality and byte/format properties required by a test with approved transforms; a hash does not automatically anonymize a person. Omit unneeded payloads and issue fresh workspace credentials. Never copy production refresh tokens, keys or notification destinations into a dev fixture.

Recorded content is untrusted input, including text later shown to a coding agent. It cannot authorize commands, reveal secrets or change acceptance rules. An execution manifest verifies destinations and principal scope independently of captured headers or environment names.

Capsules have access scope, transform/policy version, retention and revocable references. Pinning preserves a useful case within policy; it does not mean permanent undeletable data. Deletion/suppression must reach bases, raw artifacts, catalogs and historical snapshots. Keep production serving during capture failure, but block unsafe export/effects and incomplete verification from passing.

Send, retry and reconnect

This is the textbook's illustrative test, not a claimed production incident.

Start with one sanitized base containing two fresh workspace principals, a conversation and representative history. Baseline and candidate each receive isolated writable copies. Select the authenticated send → persist → realtime delivery → push-intent region; outside-world push delivery is denied or uses an approved sandbox.

1. Admit one send with logical idempotency key k.
2. Let the selected persistence operation commit.
3. Lose its acknowledgement at the defined post-commit boundary.
4. Reconnect and retry the same logical send with k.
5. Observe stored state, realtime output and push intent. Do not inject the generated downstream event a second time.

Assert one durable message, one stable message identity, one visible message and no effect outside the workspace. Transport retries may occur; product-level idempotency decides correctness. A faulty implementation yielding two rows and a fixed one yielding one are illustrative expected outcomes.

The capsule stores artifact/base/runner/assertion versions, fresh-principal references, workload/key, fault placement, causal journal, state/effect diff, trace references and coverage. Its recipe reconstructs the declared pre-state and reproduces the failed assertion. The comparator's paired observation links the two executions without merging their trace trees.

From anomaly to regression case

Admit a detected anomaly into a restricted investigation workspace, verify its source/coverage, reconstruct the minimum required state and reproduce it before promotion. Minimize inputs/state/faults only while the original assertion still fails. Keep the original evidence reference and the minimization provenance.

Add concurrency variants separately: simultaneous same-key sends, ordering changes, disconnect timing and delayed duplicate delivery. Fuzzing explores allowed domains, fault injection explores named boundaries, and a stable corpus prevents the known failure from returning. Frequency-based deduplication cannot remove a rare correctness case.

Results and evidence

Separate passed, assertion failed, policy blocked, capture incomplete, provisioning failed and measurement failed. A no-op denied dependency is not success. A missing trace or sampled zero is not an assertion pass. Results include denominator/coverage and uncertainty, with rich evidence available even before historical SQL visibility.

Required release evidence is defined in Delivery gates; client state and agent acceptance ownership are covered in Clients and agents.