Operations & Reliability
Reaktor connects production health, development experiments and release evidence through the same graph identities. Operators should be able to follow a failing user journey into its service, runtime instance, dependencies, profile and reproducible Shadow case.
Direction agreed 1 October 2026. ClickHouse continues to serve product analytics and Reaktor service evidence. The proposed Analytics Engine and R2/Iceberg path adds hot Shadow comparison and historical investigation; it does not replace those consumers. Shadow workspaces share the observability infrastructure and isolate application state, credentials, effects and measured capacity.
Capture direction extended 3 October 2026. Shadow aims to acquire everything each supported layer exposes: graph/application behavior, deep JVM evidence, kernel/eBPF activity, JavaScript instrumentation beyond proxies, Worker events/bindings, Kubernetes changes, state and clients. The Capture fabric defines this scope and the evidence needed to distinguish acquired observations from qualified replay.
Native diagnostics decision — 7 October 2026. Crashlytics remains the mobile crash collector. Firebase Performance supplies qualified mobile timing; Reaktor owns session/trace identity, explicit breadcrumbs, resource probes and a small client journal. Full Shadow reproduction and release comparison follow launch. Native diagnostics with Crashlytics defines coverage and acceptance. The legacy debug Measure integration remains in source pending removal; its hosted design is superseded.
Read by task
| Task | Start here | Then read |
|---|---|---|
| Understand current health and signal delivery | Observability | Analytics + Telemetry |
| Find the responsible graph scope and release | Graph evidence | Connected panes |
| Diagnose size, processes, threads or memory growth | Instance diagnostics | Profiling |
| Diagnose a mobile crash, ANR or slow session | Crashlytics and client evidence | Instance diagnostics, Verification |
| Reproduce behavior safely | Reaktor Shadow | Foundations, Workspaces |
| Record execution across every available layer | Capture fabric | Graph evidence, Backends |
| Qualify replay from captured evidence | Capture and replay | Backends, Verification |
| Test clients, live updates or agent changes | Clients and agents | Delivery gates |
| Establish capacity and recovery | Load and resilience | Performance overview |
Current implementation and intended direction
This inventory describes the local source tree and dated receipts. It is not a fresh production availability audit. A source implementation, a connected deployment and a qualified release are different claims.
| Area | Present evidence | Work still required |
|---|---|---|
| Shared observability | Checked-in Alloy, Prometheus, Loki, Tempo/Grafana configuration; JVM and Worker ingestion recorded on 13–14 September | Complete client ingestion, exact source/graph joins and cross-runtime propagation; recheck live health |
| Graph instrumentation | Port interceptor chain, typed attachments, semantics, telemetry plan/install/uninstall; SDK/export transport | Qualify boundary coverage; reference escapes remain gaps; export events and links faithfully |
| Delivery health | Bounded processor reports accepted/exported/dropped/failed/queued spans | Durable replay journal and retention; bounded retry/backoff; distinguish ingestion from query visibility |
| Product analytics | BestBuds Worker writes ClickHouse events; workbench reads analytics | Maintain event/privacy contracts and release attribution |
| Mobile diagnosis | Crashlytics adapters and native DevTools vitals; Measure removed from source; kernel-owned Crashlytics report reads implemented, provider readback blocked by Firebase sign-in | Qualify release reporting, symbols, graph/trace joins, provider reads and the small journal per Crashlytics; qualify timing and deeper leak analysis separately |
| Performance gates | Reports, budgets and required-measurement manifest distinguish absent/failed measurements | Connect each release's required probes and representative baselines |
| Keploy | k3s recording and R2 bundle transfer, local replay scripts | Pin runner and candidate artifact, source pre-state, effect policy and protocol conformance |
| Capture fabric | Graph hooks, Java-agent configuration, Keploy and client/instance measurement foundations | Deep JVM recorder, general host eBPF capture, stronger JavaScript instrumentation and complete Worker/Kubernetes joins |
| Moving Shadow | Design and seed/provisioning building blocks | Validated base publication, isolated workspace lifecycle, operation journal and paired verification |
| WAE / Pipelines / Iceberg | Architecture selected for evaluation | No end-to-end Reaktor implementation or qualification is claimed |
Source locations and dated receipts are indexed in References. Unknown data stays unknown in the workbench; an empty chart cannot prove zero errors.
One graph, several kinds of evidence
The architecture graph records what can connect. Runtime traces record what happened. Time-windowed metrics summarize behavior. Shadow runs record controlled experiments with their inputs, state and assertions. Profiles explain resource use. These are projections and referenced artifacts, not millions of new active application nodes.
Use existing graph ownership, Fact/TruthClass, operation receipts and evidence stores. A telemetry label does not confer authority to replay, inspect private payloads or change a deployment. The Graph evidence contract defines the joins and their coverage.
Document ownership and reconciliation
The section owns performance/observability, analytics/telemetry, instrumentation, DevTools, tooling, connected-pane diagnosis and the Shadow textbook pages. Cloud provisioning, security and graph semantics retain their existing homes and link into this section. Historical dev-environment research stays in Background and archive.
The document register accounts for the related documents, their canonical replacements and supporting roles. Existing URLs remain stable. Earlier broad designs are available as explicitly dated performance history and Shadow history.
Textbook conversion
The 25 September 2026 Reaktor Shadow design textbook is adapted into topic pages with its worked example, backend qualifications and acceptance gates. All sixteen chapters are mapped in the chapter register. Proposed type names and illustrative failures remain proposals and examples. The textbook's older source baseline is not evidence that the current deployment implements its design.