Shared Observability and Shadow Evidence
Every production service and Shadow execution uses shared signal infrastructure with explicit environment, workspace, release and access scope. Provisioning an observability stack per worktree adds cost and fragments comparisons. Application resources still need the isolation defined in Shadow workspaces.
Status, 1 October 2026: shared infrastructure and framework exporters exist; the new hot comparison and Iceberg evidence path is a design. The September ingestion receipt verifies named JVM/Worker observations at that time, not complete graph attribution or current live health.
Signal ownership and storage
| Signal | Operational purpose | Current or intended destination |
|---|---|---|
| Runtime traces, metrics and logs | Latency, failures, saturation and incident investigation | Existing OTel/Alloy → Tempo, Prometheus, Loki → Grafana |
| Product events | Funnels, cohorts, retention and recommendation inputs | Existing ClickHouse analytics, with app-owned event catalog |
| Mobile crashes, handled errors and supported ANRs | User-device diagnosis | Crashlytics; release reporting and provider reads require qualification |
| Mobile timing and application evidence | Startup/operation timing, session/trace joins and breadcrumbs | Firebase Performance where qualified; Reaktor probes and a bounded journal through the existing evidence storage contract |
| Paired Shadow observation | Candidate/baseline outcomes, normalized performance and divergence | Proposed WAE hot aggregates plus durable comparison records |
| Operation journal, state manifests and payload artifacts | Repeatable replay within declared fidelity | R2 artifacts and a transactional reference/catalog path; not sampled telemetry |
| Structured historical evidence | Scoped joins and investigation across releases/runs | Proposed Pipelines → R2 Data Catalog/Iceberg → R2 SQL |
| Profiles and build-size breakdowns | Explain resource use and regressions | Original artifacts in retained storage, normalized reaktor-performance summaries |
ClickHouse stays. A future Shadow read adapter may use it where a concrete latency, query or cost requirement justifies that path. The first Shadow slice does not require migrating any existing ClickHouse consumer.
The paired observation
The comparator emits one record for an admitted logical comparison, with a versioned schema. The following is a proposed record contract, not a shipped Kotlin class:
observation_id, schema_version, comparison_id, run_id
project_id, tenant_scope, workspace_id, operation, profile
baseline_execution_ref, candidate_execution_ref
artifact_vectors, graph_revision_refs, base_revision, policy_version
baseline_outcome, candidate_outcome, assertion_result, divergence_kind
baseline_duration_ns?, candidate_duration_ns?, measurement_quality
baseline_trace_refs[], candidate_trace_refs[], journal_ref?, capsule_ref?
coverage, missing_boundaries, capture_loss, admitted_at, completed_at
For controlled verification, both sides start from equivalent pre-state and run the same admitted inputs. An observed production execution may be attached as contextual evidence; a production request and a candidate running against today's sanitized state are not automatically a reproducible pair.
Separate execution traces retain their own parents. The comparison record relates them. Match operations inside the pair by semantic operation, resource, causal position and attempt; a join on comparison_id alone multiplies unrelated spans. Missing or ambiguous pairings remain missing.
Hot comparison in Analytics Engine
Use a small fixed schema: bounded operation/profile/outcome dimensions, numeric durations and counters, and an index chosen for the actual query cohort. Keep payloads, trace trees and unbounded touched-node sets in the evidence store. WAE has fixed datapoint limits and a single-table query model, so it is not the replay catalog or a place for arbitrary trace joins. Limits, SQL statements.
Sampling is part of the measurement. Use _sample_interval weighting for supported aggregate queries and report cohort, time range, observed records, weighted estimate and sampling/coverage quality. A sampled absence of divergence cannot prove that a rare failure is absent. Exact assertion failures and missing evidence remain authoritative run results outside the aggregate. Sampling, aggregate functions.
Durable evidence and visibility
Pipelines can persist structured records into Iceberg; rich payloads and profiler binaries stay separate R2 objects referenced by checksum and policy-controlled identifiers. Admission, durable ingestion, table commit and query visibility each have their own timestamp and failure state. The Iceberg sink rolls files at a default 300 seconds, with a minimum 60 seconds; that introduces visibility delay, not an immediate-query guarantee. Sink documentation.
A hot regression must link immediately to a run/capsule or journal endpoint even while its Iceberg rows are pending. R2 SQL is currently a read-only beta query surface with documented query limitations; bound scans by time, run and project and retain a fallback artifact read. R2 SQL limitations.
The hot write and archive write are independent unless a durable dispatcher provides a stronger contract. Assign stable observation IDs, retry idempotently where supported, deduplicate in reads and reconcile missing counterparts. Pipelines' downstream delivery guarantee does not create an atomic transaction with WAE. Pipelines.
Current exporter limitations
ReaktorSpanProcessor uses a bounded queue and counts drops. Failed exported batches are counted but are not retained as a durable journal. HttpOtlpTransport retries immediately. The current OTLP JSON serializer writes span attributes/status/timing but omits events and links. Therefore even successful archival of that representation cannot preserve full causality or replay input.
Keep production response handling independent of telemetry delivery. For verification, incomplete capture blocks a green replay result. Monitor queue depth, byte rates, drops, permission errors, last successful export, archive backlog and query visibility; exclude exporter traffic from recursive instrumentation.
Access, retention and deletion
Authenticate ingestion, enforce project/workspace scope server-side and keep ingestion credentials out of distributed clients. Classify before durable capture; raw payload access is narrower than metric access. Leases, artifact retention, pinned cases and deletion requests share an explicit policy. Pinning never overrides a legal/privacy deletion or revoked authorization.
Deleting Iceberg rows and expiring historical snapshots/files are distinct operations. Use table-aware maintenance and verify old snapshots and referenced artifacts; deleting arbitrary R2 table files can corrupt a table. Deletion guidance. Monitor orphaned references and tombstones as part of delivery operations.