Shadow Workspaces and Materialization

Adapted from textbook chapters 4–6. The following contracts describe intended behavior; the complete moving-base/workspace system is not shipped. Reuse existing Reaktor graph authority, operation receipts and artifact storage rather than introducing a parallel control plane.

Proposed contracts

Contract	Responsibility
`ShadowPlan`	Selected graph region, execution profile, capture scope, dependency policies, workload and assertions
`BaseRevision`	Sanitized state/configuration/schema, transform version and per-resource source checkpoints
`WorkspaceManifest`	Workspace/base/artifact identities, verified resource destinations, principal/authority and expiry
`WorkspaceProvider`	Prepare, verify readiness and destroy owned resources, with receipts
`OperationRecord`	Boundary identity, causality, attempt/order, argument/result artifact references and coverage
`VerificationRun`	Runner/assertion versions, admitted workload, observations, measurements and terminal result
Capsule	Reconstructible subset of pre-state, inputs, faults and assertions linked to the failure

An artifact vector includes every participating service/client hash, framework/runtime compatibility, configuration/flags, migrations, models and policy versions. A Git SHA omits dirty source and build inputs; a preview URL does not preserve a build. Store actual bundles or immutable artifact references and checksums.

Workspace lifecycle

requested → preparing → ready → running → retained
                                  └──────→ deleting → deleted

Preparation/deletion failures remain visible and recoverable. Leases, quotas, cancellation and idempotent reconciliation cover every intermediate state. Cleanup stops admission, drains bounded in-flight work, revokes credentials, removes owned resources and records any remaining resource. A partial deletion cannot be reported as deleted.

Readiness verifies resolved resources and denied production destinations; an arbitrary environment header cannot select a privileged database. Secrets are references resolved by the trusted host. A run's results distinguish pass, assertion failure, policy block, capture gap, provisioning failure and measurement failure.

Publish a base safely

Only the trusted materializer has approved production-read authority. Dev runtimes cannot fall back to production on a missing row/object. Begin with a snapshot-first workflow; introduce CDC when a real freshness requirement justifies its operating burden.

Acquire a consistent source snapshot and start checkpoint L0.
Transform under a versioned privacy/schema policy into a staging base.
Where CDC is used, apply committed changes through L1 with progress persisted together with applied effects and deduplication.
Validate integrity, relationships, schema, policy and checkpoint completeness.
Publish one immutable base and atomically advance shadow-current; retain the previous valid base on failure.

PostgreSQL logical replication has schema/sequence restrictions; it does not copy all DDL or sequence state automatically. Slots can retain WAL when a consumer falls behind. Monitor retained WAL, backlog and crash recovery; explicitly cap/reseed an unhealthy materializer. Replication restrictions, logical decoding.

Cross-store state is a checkpoint vector, not an invented global transaction. Record each source position and accepted skew. If correctness requires stronger pre-state consistency, capture an adequate operation journal or move the replay cut; otherwise label the base production-shaped.

Deletion/suppression policy survives rebuilds: retain tombstones and prevent old snapshots from resurrecting removed data. A changed sanitization policy requires a new base identity and reevaluation of retained capsules. Preserve indexes, constraints, grants and row-level authorization behavior appropriate to the test.

Per-resource isolation

Resource	Honest materialization strategy	Important boundary
Postgres	Real restore/clone or a qualified isolated database/schema with complete required state	`search_path` does not merge overlay and base rows; sequences, constraints, grants and authorization need explicit handling
R2	Approved immutable base objects plus workspace writes/tombstones through a defined object view	Define list/delete/overwrite/multipart/body behavior; never production read-through on a miss
D1 / KV	Separate seeded resources with deliberate cache/expiry state	Empty local storage is not equivalent to the selected base
Durable Objects	Isolated namespace with application-level export/restore of stored state	No claim to clone heap, live sockets, callbacks or pending alarms
Queues / workflows	Workspace-owned names/IDs and explicit source of admitted events	Execute the producer or inject its recorded event, once
Memgraph / ClickHouse	Isolated instance or verified access scope, copied state and derived-view policy	Rebuilding a projection is different from testing its original ingestion path
Clients	Seeded local DB/cache/pending state and fresh workspace sessions	No copied production refresh tokens, device keys or notification destinations

An expensive shared dev resource may be acceptable for functional work with verified isolation. It cannot support an uncontended performance claim without measuring the interference. A shared JVM hosting both trust tiers is an explicit limitation; use separate candidate processes where authority or capacity requires it.

Operating acceptance

Crash between apply and checkpoint, repeat preparation, expire a lease mid-run, revoke authority, fail cleanup and test deletion/reseed. Production changes must arrive sanitized at a validated base; running experiments must remain pinned; cleanup must leave evidence of what remains. Delivery gates define the first vertical slice.