Operations Delivery Gates and Runbooks
Adapted from textbook chapter 16 and the current-source analysis. Build one qualified journey, then expand toward the complete Capture fabric across the available runtimes and infrastructure. Each gate delivers observable behavior and retained evidence; none is satisfied by a declaration or a fixture chart.
Implementation sequence
| Gate | Deliverable | Acceptance evidence |
|---|---|---|
| 1. Safe moving dev | Approved snapshot → sanitize → validate → immutable base; isolated writable workspace and cleanup | Production change appears in a published sanitized base; denied production effects; repeat preparation/expiry/deletion receipts |
| 2. One repeatable journey | Send/retry/reconnect with equivalent baseline/candidate pre-state and fixed assertions | Deliberately introduced duplicate regression fails, fixed candidate passes, capsule repeats within declared fidelity |
| 3. Evidence transport and capture conformance | Qualified journal, layered collector matrix, graph/source/activation IDs, propagation, serializer fidelity, cleanup and coverage | Same real operation resolves across collectors; failures/loss are visible; required backend and capture cases pass |
| 4. Hot and historical comparison | Paired observation, WAE schema, durable archive, reconciliation and bounded forensic query | Known divergence appears with correct denominator; evidence opens before Iceberg visibility; dual-write failures reconcile |
| 5. Agent, clients and scale | Fixed oracle, selected graph scope, device/update journeys, real-dependency load/soak/fault profiles | Exact tested artifact meets contract/SLO/resource budgets; required missing probes block; rollback/cleanup qualified |
Snapshot-first is deliberate. Add CDC after publication and isolation work; test apply/checkpoint crashes, retained WAL and reseed before calling the base continuously fresh. Expand from one real case into protocol/storage/runtime conformance. Preserve the September instrumentation item IDs as a historical dependency register, with current status in the Instrumentation Plan.
Capture fabric gate
Qualify a JVM journey with graph/Service records, Keploy dependency exchanges, deeper agent evidence, host/kernel observations and pod/image identity. Qualify a Worker journey with handler, binding, durable-event and platform evidence. Start with the collectors available now, and explicitly mark the deeper proposed capabilities missing until they are demonstrated. Extend the same inventory across the remaining layers; a minimal replay is a first delivery slice rather than the final capture scope.
For each collector retain the exact build/version, acquisition settings, attachment scope, capabilities, filters/sampling, payload codecs, event/byte loss, durable artifact references and measured perturbation. Demonstrate one known failure, candidate replay with denied real effects, identical parallel operations, a raw path outside semantic hooks, collector failure and cleanup. Diagnostic evidence may remain useful when replay coverage is incomplete; the result must say which claim passed.
Ownership and storage
Reusable orchestration contracts belong in the Reaktor framework through existing graph plans, authority and resource ownership. Runtime-specific adapters belong in their owning modules. A proposed reaktor-shadow module is a placement proposal, not a claim that it exists. BestBuds owns its sanitization rules, schema materializers, product journeys and assertions.
Use R2 for retained bundles/profiles and the existing transactional catalog/receipt mechanism for manifests, leases and references. Do not add a new database merely for Shadow. Shared telemetry does not require shared application authority. Retain ClickHouse's analytics and service-evidence consumers while adding the new comparison path. Crashlytics remains the mobile crash collector; qualify Firebase Performance and existing resource probes separately. Deliver minimum session/trace correlation and a bounded client journal before launch. Full Shadow reproduction follows launch; ordinary production evidence contains no payload bodies.
Operate the system that gathers evidence
| Pipeline | Signals and failure actions |
|---|---|
| Materializer | Base age, per-store checkpoint/skew, backlog, WAL retention, transform/validation failures; pause/reseed safely and retain prior valid base |
| Capture | Event/byte rates, drops/truncation, incomplete sessions, unknown boundaries, rejected payloads and export lag; preserve production and mark evidence incomplete |
| Workspace | Active leases, preparing/deleting age, resource quota, orphaned destinations, credential revocation and cleanup failures; reconcile ownership before retry |
| Hot/archive ingest | Accepted/failed/deduplicated records, missing counterpart IDs, ingestion/commit/query lag and permissions; retry/reconcile without claiming atomic dual writes |
| Verification | Result classes, corpus/coverage, fixed assertion version, replay flakiness, missing collectors and generator health; block unsupported passes |
| Retention/cost | Base bytes, captured bytes, workspace deltas, pinned artifacts, scan/ingestion usage and unreferenced objects; enforce policy-aware retention and deletion |
Freshness, health and completeness are different. A fresh mirror can contain incomplete capture; a healthy exporter can have delayed query visibility. Define alerts from observed normal operation and product objectives rather than arbitrary universal thresholds.
Incident workflow
Confirm the target/release and collector health; scope the graph/time range; inspect request and instance evidence; retain the qualified profile or failure case; reproduce in a pinned isolated workspace; test the candidate against fixed assertions and budgets. Keep read diagnosis distinct from restart, scale, migration or deployment authority. Attach evidence to the existing operation receipt.
For missing telemetry, check producer emission, queue/drop counters, ingestion authorization, export response, table/trace visibility and read scope in that order. Do not create a duplicate collector/data source because a chart is empty. For suspected leaks, use the repeated-lifecycle workflow in Instance diagnostics.
Current runbook authority
The checked-in k3s observability runbook and September receipt own actual installed destinations, dashboards and credential-rotation procedures. Existing Pulumi definitions must be reconciled with that installation before reuse. This page defines the operating contract, not fresh remote health or authority to activate paid services. See implementation evidence.
Final acceptance
A production change reaches a validated sanitized base; equivalent baseline/candidate workspaces exercise a real journey; a meaningful assertion failure reproduces; measurements identify their target and quality; production remains unaffected; workspace and artifact cleanup obey policy. The workbench and agents can follow that result back to graph, source, instance and original evidence.