Performance design history

Superseded design archive, preserved 1 October 2026. The text below is the pre-rewrite document, including its dated reviews. Its current-state claims, schedules, isolation assumptions and guarantees may be obsolete. Read Operations & Reliability and the current overview for the reconciled direction.

Full-stack design for performance measurement, profiling, and production telemetry across mobile, web, desktop, server, workers, and the data layer — built on reaktor-performance, reaktor-telemetry, and the existing OpenTelemetry-KMP plumbing, with every metric attributed back to a graph node.

• Owner Reaktor framework
• First consumer BestBuds
• Drivers Maestro · Flashlight · Lighthouse · Keploy · k6 · async-profiler · Instruments · Perfetto · CDP
• Sink Grafana Tempo / Mimir / Loki
• Prod guardrails SLOs · burn-rate alerts · runbooks · telemetry governance
• Source spec Reaktor Strategic Assessment §29

1. Executive summary

Reaktor already has the right primitives — reaktor-performance ships a ReaktorPerformanceCollector with budgets, samples, vitals snapshots, flamegraph frame trees, and a ReaktorPerformanceHarness that timestamps any block and records the run; reaktor-telemetry wraps OpenTelemetry-KMP and already opens per-graph and per-node spans; reaktor-work example workers log the attempt count; a structured Work telemetry adapter remains proposed. What's missing is end-to-end wiring: the collectors are not yet emitting into OTel, the remaining platform drivers (Flashlight, async-profiler, k6, Keploy) are typed as enums but not invoked from CI — Lighthouse is now a real, standard runner (see §2.4) — and no metric crosses the boundary into Grafana.

This document is the plan to close that loop. It treats performance as a fourth telemetry signal alongside traces, metrics, and logs, all attributed to graph nodes via the same node.id tag GraphTelemetry already emits. Each platform gets a thin VitalsCollector bridge that feeds the common ReaktorAppVitalsCollector; each driver gets a HarnessRunner that wraps the upstream CLI; everything funnels through one ReaktorPerformanceReport that is both serializable (CI artifacts, PR comments) and exportable (OTel metrics → Mimir histograms).

The production gap is bigger than "more timers". Reaktor still needs an operations contract: SLOs for user journeys, alert routes that distinguish pages from tickets, runbooks tied to owners, deployment and feature-flag correlation, telemetry self-monitoring, retention and cost limits, PII redaction, cardinality budgets, and synthetics that tell us when the outside world is broken before users do.

2. Current state of reaktor-performance

2.1 What ships in reaktor-performance

2.2 Adjacent infrastructure

2.3 Gaps that block end-to-end perf

2.4 Lighthouse runner (standard)

Lighthouse is the first driver wired end-to-end. The contract — the category taxonomy, the durable lighthouse.* metric names, and the recommended budgets — lives once in ReaktorLighthouse.kt (commonMain) and is mirrored for the Node runner in reaktor-performance/ts/src/index.ts. The runner (reaktor-performance/tools/lighthouse-report.mjs) does config → run (median of N) → map LHR → ReaktorPerformanceReport → budgets → assert, so a Lighthouse audit lands in the same report/budget model as every other surface and is ready for the OTel pipeline below.

Recommended budgets (ReaktorLighthouseBudgets.recommended()), at the web.dev "good" thresholds:

The higher-is-better category-score floors are why ReaktorPerformanceBudget gained a direction (Max cap / Min floor); every existing budget defaults to Max, so nothing else changed.

# CI-safe unit — map a saved LHR fixture, no Chrome:
node reaktor-performance/tools/lighthouse-report.mjs --self-test \
  reaktor-performance/tools/fixtures/lighthouse-sample.json

# Reference wiring (BestBuds, needs Chrome; chrome-launcher honors CHROME_PATH):
npm run perf:reaktorWeb:lighthouse   # build → serve dist → audit → --assert budgets

--assert exits non-zero on any Error-severity violation, mirroring requireWithinBudgets(). The perf:reaktorWeb:playwright field-vitals spec stays the unthrottled lab proxy; ReaktorLighthouseBudgets is the canonical CWV source enforced against a throttled audit here.

3. Guiding principles

1 · Graph attribution is sacred

Every metric, span, and log carries node.id (and graph.id for nested graphs). The visitor-walker pattern already established by GraphTelemetry is the precedent: instrumentation lives outside the node, not inside.

2 · One collector, many sources

ReaktorPerformanceCollector stays the single in-process aggregator. Platform code feeds it; OTel exporters drain it. The harness is the CI driver; production telemetry is the runtime driver. Same model both ways.

3 · Don't measure what you can't act on

Every metric ships with a budget (ReaktorPerformanceBudget) and a severity (Info/Warning/Error). A metric needs a defined failure threshold before collection. Error blocks CI; Warning annotates the PR; Info goes to the dashboard.

4 · Drivers are upstream tools, not reinventions

The proposed HarnessRunner wraps async-profiler, Lighthouse, Flashlight, Keploy, k6, Instruments. The HarnessRunner normalises their output into ReaktorPerformanceToolRun + ReaktorPerformanceMetric; everything downstream is uniform.

5 · OTel is the only wire format

Traces → Tempo, metrics → Mimir, logs → Loki, vitals → Mimir as histograms. No bespoke protocol, no second SDK. The W3C trace-context header threads request IDs from web/mobile → CDN → server → DB.

6 · Production and CI share the schema

The same ReaktorPerformanceReport a Maestro run produces in CI is the same shape Mimir queries return for a production window. PRs are diffable against last-week production. Promotes "this regresses prod" from instinct to query.

7 · SLOs decide alert severity

Raw thresholds are not enough. Pages are driven by error-budget burn on user journeys; tickets are driven by component degradation. A slow RepositoryNode matters when it threatens login, chat send, media upload, or notification delivery.

8 · Telemetry must be observable

The collector, exporters, samplers, and storage backends emit their own health metrics: dropped spans, queue depth, ingest latency, rejected series, and monthly projected cost. "No data" is treated as an alertable state, not as calm.

4. The unified data model

Three signal types plus a derived dashboard signal — every reaktor-performance type already in the codebase fits one of these four:

Resource attributes attached to every signal:

service.name        = "bestbuds-android" | "bestbuds-ios" | "bestbuds-web" | "reaktor-server" | "chat-worker" | ...
service.version     = git sha
deployment.environment = "dev" | "staging" | "prod"
reaktor.module      = "reaktor-graph" | "reaktor-auth" | ...
graph.id            = root graph UUID
graph.label         = "BestBuds" | "ChatGraph" | ...
service.instance.id = pod name | worker isolate id | device install id hash
deployment.release = git sha + build number + remote-config version
cloud.region       = "bom" | "lhr" | "iad" | ...
app.build          = Android versionCode | iOS CFBundleVersion | web build id
device.class       = "low" | "mid" | "high"
network.type       = "wifi" | "cellular" | "offline"
user.segment       = "free" | "paid" | "internal" | "synthetic"  // never raw user id
tenant.hash        = stable hash when multi-tenant; no tenant name in labels

// span/metric specific:
node.id             = RouteNode/ServiceNode/ActorNode UUID
node.type           = "RouteNode" | "ServiceNode" | "ActorNode" | "RepositoryNode" | ...
node.label          = human label
route.pattern       = "/chats/{id}"   // for routes
transport           = "HTTP" | "LOCAL" | "PEER" | "PUBSUB" | "QUEUE" | "WORKFLOW"  // for services

5. Stack architecture

┌────────────────────────────── Sources (where measurement happens) ──────────────────────────────┐
│ CLIENT                         SERVER & WORKERS                  DATA                           │
│ ┌─────────────────────────┐    ┌────────────────────────────┐    ┌────────────────────────┐     │
│ │ Android FrameMetrics    │    │ ServiceInterceptor span    │    │ pg_stat_statements     │     │
│ │ iOS CADisplayLink       │    │ async-profiler JFR         │    │ Hyperdrive metrics     │     │
│ │ Web PerformanceObserver │    │ DbAdapter query span       │    │ Memgraph SHOW STATS    │     │
│ │ Desktop JFR             │    │ Workers Analytics Engine   │    │ SQLDelight query span  │     │
│ │ Lighthouse / Flashlight │    │ WorkerBase duration/retry  │    │                        │     │
│ └────────────┬────────────┘    └─────────────┬──────────────┘    └────────────┬───────────┘     │
└──────────────┼───────────────────────────────┼────────────────────────────────┼────────────────┘
               ▼                               ▼                                ▼
┌────────────────────────────────────────────────────────────────────────────┐
│ ReaktorPerformanceCollector (one per process)                             │
│ ┌──────────────────────────┐ ┌──────────────────────────┐                 │
│ │ ReaktorAppVitalsCollector│ │ ReaktorPerformanceHarness│ (CI)            │
│ └──────────────────────────┘ └──────────────────────────┘                 │
└──────────────────────────────────────┬─────────────────────────────────────┘
                                       ▼
┌────────────────────────────────────────────────────────────┐
│ OpenTelemetry SDK + OTLP gRPC/HTTP exporter                │
└─────────────────────────────────┬──────────────────────────┘
                                  ▼
┌────────────────────────────────────────────────────────────┐
│ OTel Collector (k3s sidecar)                               │
│ · tail-based sampling · batch · attribute filter            │
└────────┬─────────────────────┬────────────────────┬────────┘
         ▼                     ▼                    ▼
   Tempo (traces)        Mimir (metrics)       Loki (logs)
         │                     │                    │
         └─────────────────────┼────────────────────┘
                               ▼
Grafana · auto-built dashboards per service / route / actor / worker
                               │
                               ▼
Blueprint editor overlay · same data, rendered on the graph canvas

Every box on the right of the collector is operational infrastructure; everything on the left is application code. The collector is the only thing the framework owns end-to-end.

6. Android profiling & vitals

6.1 Vitals collection

Frame metrics

androidx.metrics.performance:JankStats hooks Window and reports per-frame timing. A new AndroidVitalsBridge in reaktor-performance/androidMain subscribes, classifies each frame against SlowFrameMs/FrozenFrameMs, and pushes ReaktorFrameVitals into the collector. Window-attribution maps to the current top RouteNode via graph.backStack.top.

Startup & cold start

androidx.startup:startup-runtime hands us Process.getStartUptimeMillis(). The bridge marks ReaktorAppVitalMarks.ProcessStart, AppStarted, FirstFrame, FirstInteractive, producing ReaktorAppStartupPhase spans for each leg.

Long tasks

Custom ChoreographerFrameCallback instrumented at Looper.getMainLooper() records main-thread blocks > 50 ms as ReaktorLongTaskVitals. Stack traces sampled at the time-of-detection get attached to the OTel span.

Memory

Periodic (~30 s) Debug.MemoryInfo + Runtime.totalMemory()/freeMemory() snapshots → ReaktorMemoryVitals. Leak detection runs LeakCanary in debug builds and forwards RetainedInstanceFound events to the collector with the leak class name.

6.2 Profiling

6.3 Low-level / debugging

• adb shell dumpsys — gfxinfo (jank breakdown), meminfo (PSS), batterystats (per-UID battery), procstats (longitudinal CPU/mem).
• adb logcat piped through filterspec — Kermit lines tagged with node.id filter into the IDE.
• lldb / Android Studio native debug — for JNI / NDK code in reaktor-ffi (libreaktor, Highway SIMD).
• strace / bionic libc tracing — for syscall-level investigations on flagged hot paths.

7. iOS profiling & vitals

7.1 Vitals collection

Frame metrics

CADisplayLink attached to the main run loop in DarwinVitalsBridge. Per-frame delta classified against the same SlowFrameMs/FrozenFrameMs constants. Attribution via Reaktor's UIViewController wrapper which observes graph.backStack.top.

MetricKit

iOS 13+ MXMetricManager ships daily app-vitals reports: launch time, hang rate, CPU/memory, disk writes. DarwinVitalsBridge receives the payload via MXMetricManagerSubscriber and emits one ReaktorServerVitalsSnapshot per period — same shape as server-side, just labelled as client.

Cold start

DYLD_PRINT_STATISTICS in debug, MXAppLaunchMetric in release. Pre-main + main + first-frame phases each become ReaktorAppStartupPhase entries.

Memory

mach_task_basic_info (resident_size) sampled every 30s; os_signpost markers around large allocations. Combined with MetricKit's MXMemoryMetric for peak/average.

7.2 Profiling

7.3 Low-level / debugging

• lldb for crash post-mortem; image lookup --address <pc> against the shipped dSYM.
• Console.app / system log with subsystem filter dev.shibasis.reaktor — set by every Kermit log on iOS via the OSLogWriter.
• sysdiagnose for full system snapshots when reproducing a customer issue locally.

8. Web vitals & Lighthouse

8.1 In-browser collection

// reaktor-performance/jsMain/.../WebVitalsBridge.kt
@JsExport
object WebVitalsBridge {
    fun install(collector: ReaktorPerformanceCollector) {
        // Uses web-vitals package (Google) for canonical LCP/INP/CLS/FCP/TTFB.
        onLCP { metric -> collector.recordVital("LCP", metric.value, attribution(metric)) }
        onINP { metric -> collector.recordVital("INP", metric.value, attribution(metric)) }
        onCLS { metric -> collector.recordVital("CLS", metric.value, attribution(metric)) }
        onFCP { metric -> collector.recordVital("FCP", metric.value, attribution(metric)) }
        onTTFB{ metric -> collector.recordVital("TTFB", metric.value, attribution(metric)) }

        // Long tasks
        PerformanceObserver({ list ->
            list.getEntries().forEach { collector.recordLongTask(it.duration) }
        }).observe(jso { type = "longtask"; buffered = true })

        // Memory (Chrome only — performance.memory)
        window.setInterval({ collector.recordMemory(performance.memory) }, 30_000)
    }
}

Attribution: each web-vitals callback's attribution field gives the LCP element / INP target. We map this DOM node to the RouteNode by storing data-reaktor-node-id on every container that wraps a route — added automatically by the existing WebNavigationBridge.

8.2 Lighthouse CI

Lighthouse runs once per PR against a preview deployment (reaktor.build for the docs site, a Cloudflare preview URL for the BestBuds app). The harness wraps it:

// reaktor-performance/jvmMain/.../LighthouseRunner.kt
class LighthouseRunner(val target: String) : HarnessRunner {
    override fun run(scope: ReaktorPerformanceScope): ReaktorPerformanceToolRun {
        val report = exec("lhci collect --url $target --numberOfRuns=3")
        val parsed = LighthouseJson.parse(report)
        return ReaktorPerformanceToolRun(
            name = "lighthouse",
            tool = ReaktorPerformanceTool.Lighthouse,
            metrics = listOf(
                metric("performance", parsed.categories.performance.score, domain = WebVitals),
                metric("LCP",         parsed.audits["largest-contentful-paint"].numericValue),
                metric("TBT",         parsed.audits["total-blocking-time"].numericValue),
                metric("CLS",         parsed.audits["cumulative-layout-shift"].numericValue),
                metric("SI",          parsed.audits["speed-index"].numericValue),
            ),
            reportPath = "lighthouse-report.html",
            ...
        )
    }
}

8.3 Web bundle budgets

Vite emits a JSON manifest per build. A small Gradle task in :reaktorWeb reads it and builds ReaktorBuildArtifactMetrics per entry chunk. Budgets: < 200 KB gzip initial, < 50 KB per lazy route. Block merge if initial > 220 KB.

9. Desktop (JVM) profiling

Compose Desktop runs on the same JVM as the server-side, so the same tools apply. The differences are in what we capture:

• JFR (Java Flight Recorder) always on, low-overhead, with a reaktor.jfc template (custom events for Graph.dispatch, port writes, navigation). Stored in a circular buffer; on crash or budget violation, the buffer dumps to ~/.reaktor/jfr/<timestamp>.jfr.
• async-profiler attached via the JVM agent for on-demand wall-clock and allocation profiling. Triggered by the dev tools panel (already exists in reaktorDesktop).
• Skiko render stats — RenderStats from Compose Multiplatform's Skia layer fed into ReaktorFrameVitals.
• JVisualVM / IntelliJ profiler for ad-hoc dev profiling.

10. Maestro · Flashlight · Playwright

Three different drivers, one normalised output via the harness:

M

Maestro — UI flow regression with timing assertions

bestbuds/maestro/ already has flows. Each step can include assertVisible + a new extendedWaitUntil timeout that becomes a soft budget. MaestroRunner parses Maestro's JUnit XML output and emits one ReaktorPerformanceToolRun per flow, with metrics for per-screen transition time, dropped frames during scroll (via maestro --record-trace Perfetto export), and memory delta.

F

Flashlight — "Lighthouse for mobile"

npx @perf-profiler/cli flashlight measure --duration 30 against a USB-connected device while running a Maestro flow. Captures CPU, RAM, FPS, JS thread (for any React Native parts), per-process detail. Output JSON parsed by FlashlightRunner into ReaktorAppVitalsSnapshot. Score < 80 fails CI.

P

Playwright — web e2e with trace + perf

page.route('**/*', ...).waitForLoadState('networkidle') plus page.evaluate(() => performance.getEntries()) at the end of every test. Trace.zip is uploaded as a CI artifact. PlaywrightRunner aggregates ResourceTiming entries into a network waterfall ReaktorPerformanceMetric list.

11. JVM server profiling

11.1 Continuous profiling in production

JFR runs continuously on the Spring Boot reaktorServer with the reaktor.jfc profile (CPU sampling at 19 ms, allocation sampling, lock contention, file I/O). Files rotate hourly; jfr-uploader sidecar ships them to S3 (with lifecycle: 7 days hot, 30 days cold). A separate "always-on" async-profiler instance runs in cpu+alloc mode at 999 Hz, output streamed to Pyroscope (single binary, deployable on k3s) for live flamegraph queries by node UUID.

11.2 Service interceptor

The existing ServiceInterceptor (reaktor-graph/.../service/ServiceInterceptor.kt) gets a new OtelServiceInterceptor implementation that opens a span per request:

class OtelServiceInterceptor(private val tracer: Tracer) : ServiceInterceptor {
    override suspend fun intercept(req: Request, chain: ServiceChain): Response {
        val span = tracer.spanBuilder("${req.handler}.${req.method}")
            .setAttribute("node.id",   req.handler.node.id.toString())
            .setAttribute("node.type", req.handler.node::class.simpleName!!)
            .setAttribute("transport", req.transport.name)
            .setAttribute("payload.bytes", req.payloadSize.toLong())
            .startSpan()
        return try {
            withSpan(span) { chain.proceed(req) }.also {
                span.setAttribute("response.bytes", it.payloadSize.toLong())
                span.setStatus(if (it.isError) ERROR else OK)
            }
        } finally { span.end() }
    }
}

11.3 Worker / actor metrics

The Work wrappers and snapshot API named below are proposed instrumentation seams, not current reaktor-work symbols. The Work roadmap owns their final contracts and metric names.

11.4 Thread & lock analysis

Coroutine names — CoroutineName(node.id.toString()) set by Graph.coroutineScope. Async-profiler's --threads output then groups by coroutine, and Pyroscope flamegraphs colour by node. BlockHound in tests (not prod) catches accidental blocking calls inside Dispatchers.IO-restricted scopes.

12. Cloudflare Workers profiling

Workers have a different profiling model — no JVM, no agents, isolate-per-request. The toolkit:

• Workers Analytics Engine — built-in time-series ingest, ~25 ns write. Every handler logs { name, duration, status, region, node.id } rows. Queries via SQL API → exported to Mimir hourly by a scheduled Worker.
• Workers Logs (Logpush) — JSON logs streamed to R2 bucket, then to Loki via promtail.
• Trace events — console.timeLog(spanName) wrapped by a small tracedHandler() helper; spans exported via otlp-trace-exporter through the standard fetch API to the OTel collector.
• CPU time limits — Workers have a configured per-request CPU budget. The WorkerBudget assertion emits a Warning before the budget is threatened and an Error before the platform limit is reached.
• Bundle analysis — wrangler deploy --dry-run --outdir dist + esbuild --metafile → bundle-size budget per worker. Currently 9 workers (chat, messaging, social, media, notification, config, analytics, bot, sticker), each with its own < 1 MB budget.

The Durable Object actors get the same treatment as JVM actors above: per-object storage_ops, websocket.connections, alarm.duration metrics.

13. k6 · Gatling · Keploy

K6

k6 — scripted load tests

Reusable scenarios live beside the target they pressure, for example bestbuds/tests/reaktorServer/k6/, and run through ../reaktor/tools/k6/run-k6.sh. They surface as npm run k6:<target>:* and target-scoped reaktor <target> k6 ... commands, matching the target-first Maestro layout. k6 run --out experimental-prometheus-rw=... streams metrics straight to Mimir during the run, so Grafana panels are live. K6Runner in the future harness should wrap the same scripts; assertions such as p(95)<200ms become budgets.

The first concrete slice is auth:

npm run k6:reaktorServer:auth:smoke
npm run k6:reaktorServer:auth:load
K6_RPS=25 K6_DURATION=10m npm run k6:reaktorServer:auth:load

Karate owns API contract correctness for the same routes; k6 owns load, spike, soak, and SLO regression pressure.

GT

Gatling — same role, JVM ecosystem

Used when k6's scripting (JS) is awkward — e.g., for the JVM server's gRPC endpoints. Same wrapper pattern.

KP

Keploy — capture-replay regression

reaktor-shadow (planned, per spec §29.4) captures production Service-layer traffic. Keploy generates test suites from captures. On every PR, the suite replays against the branch; KeployRunner compares response shape + P95 latency vs baseline; > 20% degradation flags the PR.

14. Postgres & Hyperdrive

BestBuds workers hit Postgres via Cloudflare Hyperdrive (current wrangler.json shows two hyperdrive bindings: supabase + supabasedirect). Performance instrumentation:

15. Memgraph / Neo4j

The server already uses memgraphDriver() (reaktor-db/jvmMain/.../GraphDb.kt); resolvers in MannaKnowledgeGraphGraphQl.kt already time queries but return the timings as GraphQL response fields — not metrics. The change:

• Wrap the driver — OtelGraphDb decorator opens a span per Cypher query: cypher.statement, cypher.bookmarks, cypher.records.read, node.id of the resolver.
• Memgraph metrics — SHOW STORAGE INFO + SHOW REPLICATION ROLE + SHOW METRICS INFO (Memgraph 2.11+) scraped every 30 s, written as memgraph.queries.executed_total, memgraph.snapshot.size_bytes, memgraph.replication.lag_ms.
• Query plan inspection — PROFILE <query> for hot queries surfaced by pg-equivalent percentile sampling.
• Bolt-level tracing — Neo4j's Java driver supports BoltConnectionListener; we add one that emits per-connection RTT + bytes-on-wire metrics.

16. Client SQLite (SQLDelight)

SQLDelight already generates strongly-typed query objects. A TracingSqlDriver decorator around the existing SqlAdapter:

class TracingSqlDriver(private val inner: SqlDriver, private val tracer: Tracer) : SqlDriver by inner {
    override fun executeQuery(id: Int?, sql: String, ...): QueryResult<R> {
        val span = tracer.spanBuilder("sqlite.query")
            .setAttribute("db.statement", sql)
            .startSpan()
        return try { inner.executeQuery(id, sql, ...) }
        finally { span.end() }
    }
}

On client, spans are sampled at 1% in production (mobile bandwidth is expensive) but always exported in dev/debug builds. SqlAdapter.checkSize() (already exists) becomes a periodic gauge: sqlite.db.bytes per database.

17. OTel pipeline · Tempo · Mimir · Loki

app process                 k3s cluster                 query
─────────────               ────────────────            ─────────
OTel SDK ── OTLP/HTTP ──▶ OTel Collector ──┬──▶ Tempo ──▶ Grafana traces
(per process)             (DaemonSet)      │
                                           ├──▶ Mimir ──▶ Grafana metrics
                                           │
                                           └──▶ Loki ──▶ Grafana logs
                                             ▲
                                             │ Cloudflare Workers via fetch(otlp endpoint)
                                             │ + Logpush for logs
                                             │
                                             └── Postgres scraper · Memgraph scraper · CDN logs

17.1 Sampling

• Head sampling at the SDK: 100% in dev, 10% in staging, 1% in prod by default.
• Tail sampling at the collector: always keep traces with span.status=ERROR or duration > p95 budget. Drops noise but preserves the cases worth investigating.
• User-bound sampling: 100% of traces for any session with a non-null debug.session_id header — set by the dev tools panel for "trace this user" debugging.

17.2 Backend choice

Grafana LGTM stack (Loki + Grafana + Tempo + Mimir) is the default — all open-source, deployable on the existing k3s, queryable from one Grafana instance. Alternative for "I don't want to operate this": Grafana Cloud free tier (50 GB logs, 50 GB traces, 10 k series). Spec aligns with Grafana stack; this design keeps the option open.

Consolidation option: once ClickHouse lands on k3s as the analytical/OLAP tier (for product analytics), it can also back all three telemetry signals — logs, traces, and metrics — in one columnar store (Grafana queries it via the ClickHouse datasource), collapsing the three LGTM stores into one. Workers Analytics Engine stays the no-ops edge counter store either way. Worth it only when you're already operating ClickHouse for analytics; until then, LGTM is the lower-friction default.

17.3 Telemetry pipeline health

The observability pipeline needs its own dashboard and alerts. Otherwise a healthy-looking system can simply be a silent exporter. The OTel Collector, app-side exporters, and backend receivers emit these minimum self-metrics:

17.4 Exemplars and correlation IDs

Every histogram that represents latency or frame time stores exemplars with trace.id when the event is sampled. Grafana panels should click from service.latency_p95_ms or route.transition_ms directly into a Tempo trace. Crashlytics and MetricKit payloads attach the same trace.id, session.id_hash, deployment.release, and node.id when present.

18. Production SLOs & alerts

Production monitoring should be framed around user journeys first and framework components second. Component metrics explain incidents; journey SLOs decide whether humans are paged.

18.1 Journey SLOs

18.2 Alert policy

18.3 Burn-rate alerts

Use multi-window burn-rate alerts instead of single-threshold latency pages. Example for a 99.5% monthly login SLO:

# Fast burn: page now
login:error_budget_burn_rate{window="5m"}  > 14
and
login:error_budget_burn_rate{window="1h"}  > 14

# Slow burn: page/ticket depending on hour
login:error_budget_burn_rate{window="30m"} > 6
and
login:error_budget_burn_rate{window="6h"}  > 6

# Trend: ticket
login:error_budget_burn_rate{window="2h"}  > 3
and
login:error_budget_burn_rate{window="1d"}  > 3

18.4 Runbook shape

Every page links to a short runbook with the same structure: owner, user impact, rollback lever, feature flag lever, first five Grafana queries, known noisy dependencies, and "safe to ignore if" conditions. Reaktor-generated runbooks can be derived from the graph: route → service edge → repository → worker queue → external dependency.

18.5 Production coverage matrix

19. Telemetry governance

Telemetry needs product-grade rules before it is enabled broadly in production. The framework should make the safe path the default for every app that consumes Reaktor.

19.1 Cardinality contract

Route labels always use patterns (/chats/{id}), not concrete paths. SQL and Cypher labels use stable hashes plus parameterized statement templates. High-cardinality details stay in sampled traces or structured logs with retention controls.

19.2 Redaction and consent

• Collector redaction runs before enqueue/export and strips disallowed attributes from metrics, logs, profiles, crash breadcrumbs, and tool reports.
• Header allowlist only permits trace context, environment, app build, request id, and synthetic/debug session headers. Everything else is dropped or hashed.
• DB statement sanitizer rejects spans that contain quoted literals where placeholders should be present.
• Client opt-out disables non-essential RUM export while keeping crash-free and coarse health counters where legally permitted.
• Profile safety strips argument values from sampled stacks and stores raw profiles with shorter retention than aggregate metrics.

19.3 Metric catalog and schema tests

Add a checked-in metric catalog under reaktor/reaktor-performance/catalog/. Every metric declares owner, type, unit, label allowlist, default budget, retention tier, and dashboard panel. CI fails when a new metric is emitted without a catalog entry or when a label exceeds its declared cardinality budget in tests.

metric: service.latency_ms
type: histogram
unit: ms
owner: reaktor-io
labels: [service.name, deployment.environment, route.pattern, transport, status.class]
forbidden_labels: [user.id, request.path, db.statement]
retention: 30d raw, 180d p50/p95/p99 aggregate
alerts: service_slo_burn, dependency_latency_regression

19.4 Retention and cost controls

19.5 Release and flag annotations

Every deployment, app release, wrangler deploy, Gradle publish, feature-flag change, remote-config change, DB migration, and dependency upgrade writes a Grafana annotation and emits deployment.change. Dashboards default to overlaying these markers on latency, error rate, crash-free rate, queue lag, and bundle size.

20. CI integration & budgets

20.1 Per-PR job sequence

1. Build (Gradle, parallel for Android / iOS / JVM / JS)
2. Unit + commonTest (kotlin.test, Maestro semantics tests)
3. Bundle size measurement        →  ReaktorBuildArtifactMetric
4. Lighthouse on preview URL      →  LighthouseRunner
5. Maestro flows (changed UI)     →  MaestroRunner    + Flashlight on Android
6. Playwright (web e2e)           →  PlaywrightRunner
7. k6 smoke load (server, 1×)     →  K6Runner
8. Keploy regression replay       →  KeployRunner
9. Aggregate ReaktorPerformanceReport
10. requireWithinBudgets()        ←  fails the PR on any Error
11. PR comment with deltas vs main + links to flamegraphs / traces

20.2 Headline budgets

20.3 PR comment format

📊 Reaktor Performance Report

✅ APK size       14.2 MB    (-180 KB vs main)
⚠️  Cold start     1180 ms   (+92 ms — investigating Compose init)
✅ /chats trans.  88 ms      (within budget)
🔴 Service P95   /api/feed   410 ms    (was 180 ms on main, +128%)
   → flamegraph: new FeedRankingNode dominates at 12ms vs 2ms
   → trace example: tempo://abc-123-def

Lighthouse:  perf 88 (-3)   LCP 2.1s    INP 180ms   CLS 0.05
Flashlight (Pixel 6): score 84 (-1)    fps 58.4    jank 4.1%

21. Blueprint overlay

The Reaktor Blueprint editor (in reaktorDesktop and the visual workbench) becomes the consumption surface for everything above. Each graph node gets four overlays, switchable from the Performance tab:

Implementation: a new PerformanceOverlayVisitor in reaktor-graph/visitor/ walks the live graph and fetches the latest snapshot per node from an in-memory cache (refreshed every 5 s from Mimir). Reuses the StructuralSelector + RoutingSelector infrastructure that already exists.

22. AI-assisted analysis

Spec §29.13 calls for AI-assisted optimisation. Three concrete hooks:

Hotspot summariser

After every CI build, an MCP tool feeds the top 5 flamegraph frames + the diff to a Claude/Gemini call. Returns a markdown summary identifying which node regressed and why. Posted as a PR review comment.

Bundle bloat detector

On APK size increase, agent receives the --analyzer output (Android Studio APK Analyzer JSON) + the dependency diff. Suggests: "androidx.compose.material:material-icons-extended added 2.4 MB. Use specific imports from material-icons-core instead."

Slow-query rewriter

For any pg_stat_statements entry exceeding budget, agent receives the query + EXPLAIN ANALYZE + the schema. Proposes an index, a rewrite, or a Memgraph migration. Drops a draft PR.

None of these run autonomously — every suggestion is a draft PR or PR comment, reviewed by a human. The agent's job is to compress investigation time, not to merge changes.

23. BestBuds wiring

Concrete additions to BestBuds for the first end-to-end slice:

24. Phased plan

Phase 1~3 days

Wire the collector to OTel

OTel SDK initialisation per platform; ReaktorPerformanceCollector.export(otel); OtelServiceInterceptor registered on the JVM server; OTel collector + Tempo + Mimir + Loki deployed on dev k3s; one Grafana dashboard auto-generated per service. Definition of done: a tap in the Android app produces a trace visible in Tempo.

Phase 2~2 days

Client vitals

AndroidVitalsBridge + DarwinVitalsBridge + WebVitalsBridge. Per-screen LCP/INP/CLS on web; per-screen jank on mobile. Routes annotated with data-reaktor-node-id on web. Definition of done: Grafana shows P75 LCP per route for the docs site and per screen for the Android app.

Phase 3~2 days

DB & worker spans

OtelPgInterceptor; OtelGraphDb; TracingSqlDriver; WorkerBase wrapped with duration histogram + retry counter; pg_stat_statements + Memgraph SHOW STATS scrapers. Definition of done: a single trace shows app → server → Memgraph end-to-end with timing per hop.

Phase 4~3 days

Production guardrails

Journey SLO definitions, burn-rate alert rules, runbook templates, deployment/feature-flag annotations, telemetry self-monitoring, metric catalog, redaction tests, and retention defaults. Definition of done: a synthetic login failure creates a SEV2 alert with a runbook link, while an exporter outage creates a separate "observability blind" alert.

Phase 5~3 days

CI drivers

LighthouseRunner, MaestroRunner, FlashlightRunner, PlaywrightRunner, K6Runner, KeployRunner. The new perf.yml GitHub Actions workflow. PR-comment posting. Definition of done: a deliberate +500 KB APK regression PR fails CI; a deliberate +100 ms cold-start regression posts a Warning comment.

Phase 6~2 days

Continuous profiling

JFR always-on in production server; async-profiler → Pyroscope; per-coroutine attribution via CoroutineName(node.id). Definition of done: Grafana Pyroscope panel shows live CPU flamegraph filterable by node.id.

Phase 7~2 days

Blueprint overlay

PerformanceOverlayVisitor; four overlay modes (Vitals/Latency/Hotspot/Traces); click-through to Grafana. Definition of done: a clearly hot node in the BestBuds graph renders red on the Blueprint editor when viewed against a load-test window.

Phase 8~2 days

AI-assisted analysis

MCP tool exposing get_flamegraph(node, window), get_slow_queries(window), compare_builds(a, b). PR review bot. Definition of done: a regression PR receives an actionable comment naming the culprit node + suggested fix within 5 min of CI completion.

25. Risks & open questions

Risks

• OTel agent overhead on mobile. The JVM OTel SDK is JVM-only; on KMP we use the kotlin-api + a custom OTLP exporter built on Ktor. Need to measure that the exporter itself doesn't eat > 1% CPU or > 5 MB RSS on a Pixel 6 mid-tier baseline.
• Workers OTLP egress costs. Every span is an HTTP request out of the Worker; at scale this could match the worker's CPU budget. Mitigation: batch up to 50 spans per flush, hard 30 ms CPU cap per flush, drop on overflow with a counter.
• Pyroscope storage at scale. Continuous profiling at 999 Hz on a multi-pod JVM server produces gigabytes/day. Start with 7-day retention; revisit when prod traffic crosses 10 RPS sustained.
• iOS MetricKit lag. MXMetricManager reports are daily — fine for trends, useless for "did this PR regress cold start?" Mitigation: pair with synthetic Xcode-Cloud cold-start runs on every PR.
• Tail-sampling correctness. If the OTel collector tail-samples on duration, traces that "look fast" but include an interesting error spans can be dropped. Mitigation: explicit "always keep errors" rule, plus a sample-by-fingerprint rule for new endpoints.
• PII in db.statement. Span attributes must use parameterised SQL, never substituted values. The interceptor enforces this — add a unit test that asserts ?/$1 placeholders appear in any captured span.
• Metric cardinality blow-ups. node.id, concrete paths, query strings, or tenant identifiers can create unbounded series. Mitigation: metric catalog allowlists, collector-side label dropping, and CI cardinality tests.
• Alert fatigue. Component threshold alerts can page before users are affected. Mitigation: page only on journey SLO burn or data-loss risk; route component-only degradation to tickets unless it predicts SLO burn.
• Synthetic false positives. External probes can fail because of probe-region or vendor issues. Mitigation: require quorum across regions for pages, and label synthetic failures separately from real-user SLOs.
• CI flakiness from network-bound drivers. Lighthouse and load tests can flake on shared runners. Mitigation: 3 runs, median taken; CI cluster has dedicated perf-runners with pinned hardware.

Open questions

• Do we self-host Grafana LGTM on the existing k3s, or use Grafana Cloud's free tier? LGTM is more work to operate but keeps data sovereign.
• Is reaktor-shadow (the capture/replay component the spec calls out for Keploy) in scope here, or a separate workstream? Currently treated as a dependency this design assumes will exist.
• Should the Blueprint overlay live in reaktor-graph (with ContainerNode-aware visitor) or stay in the workbench module? Visitor lives in graph; UI overlay lives in workbench.
• How do we expose performance traces to app developers building on Reaktor who don't run our Grafana? Likely: a public https://perf.reaktor.build with read-only Grafana per project, gated by Reaktor account.
• Who owns each production SLO: framework owner, product team, or service owner? The first version assigns primary owners in the SLO table, but escalation policy needs named rotations.
• What is the default RUM sampling rate for BestBuds by region and app build? The privacy and cost answer should be decided before enabling production export.
• Does telemetry require region-specific storage for user data residency, or is hashed/coarse telemetry acceptable for the first production slice?
• For the AI-assisted analysis hooks — do we run an in-cluster MCP server or call out to Claude API? In-cluster keeps data sovereign but adds operational burden; lean toward Claude API for v1.

Source files referenced: reaktor/reaktor-performance/src/commonMain/.../{ReaktorPerformanceCollector, ReaktorAppVitalsCollector, ReaktorPerformanceBudgets, ReaktorPerformanceHarness, ReaktorPerformanceModels}.kt; reaktor/reaktor-performance/src/jvmMain/.../ReaktorPerformanceReports.jvm.kt; reaktor/reaktor-performance/src/jsMain/.../ReaktorWebVitals.kt; reaktor-telemetry/.../{TelemetryAdapter, AnalyticsAdapter, GraphTelemetry, AnalyticsSink}.kt; reaktor-graph/.../service/{ServiceInterceptor, ServiceNode, Request, Response}.kt; reaktor-graph/.../visitor/Visitor.kt; reaktor-db/.../adapters/SqlAdapter.kt; reaktor-db/jvmMain/.../db/graph/GraphDb.kt; reaktor-work/.../TaskManager.kt; proposed: reaktor/reaktor-performance/catalog/, bestbuds/cloud/k3s/observability/{alerts,runbooks}/; BestBuds: modules/app/.../analytics/{BestBudsEvents, HttpAnalyticsSink, AnalyticsService}.kt; targets/reaktorServer/.../{ServerApplication, MannaKnowledgeGraphGraphQl}.kt; targets/{chat,messaging,...}Server/src/index.ts; Spec: Reaktor Strategic & Technical Assessment, April 2026 — Chapter 29.

26. Verified state, 13 September 2026

A source-level sweep of both repositories, run to answer how far are we rather than what is planned. Section 2 records intent; this section records what a build actually contains. Where the two disagree, §2 has been corrected in place.

26.1 The pattern

The model is well ahead of the instrument, and every real instrument in the framework lives inside reaktor-flexbuffer.

reaktor-performance is 1,546 lines and is a report container, not an instrument. collector.sample(name, iterations, medianMs, bestMs, worstMs) takes the numbers as arguments. ReaktorAppVitalsCollector exposes recordFrame(duration), recordLongTask(duration) and recordMemory(...) — callers hand values in and the collector computes percentiles. The ReaktorPerformanceTool enum, naming thirteen external tools, is the tell.

That is principle 4 working as designed — drivers are upstream tools, not reinventions. It is also only half a system, and the other half is wired for one target.

26.2 What actually collects, per target

Every non-empty profiling cell is a reaktor-flexbuffer shell script.

26.3 The flexbuffer harness is the framework's profiling layer, misfiled

reaktor-flexbuffer/flamechart/ contains generate-android.sh, generate-jvm.sh, generate-js.sh, generate-native.sh, profile-ios-sim.sh, generate-all.sh, analyze.py and a written ANALYSIS.md. Alongside it: 1,146 lines of benchmark code across commonMain/jvmMain/jsMain/iosMain, a jvmBenchmark source set, a C++ bench, a TS bench, and an androidInstrumentedTest A/B.

This is the widest per-platform profiling work in either repository, and it sits inside 1 of 34 modules, in shell scripts, outside the Kotlin API, aimed at a benchmark binary rather than an application.

Corrected 13 September 2026 after review. An earlier version of this section called the harness complete, proven on real devices, and "mostly a move". Running the scripts contradicts that:

• bash -n flamechart/generate-all.sh fails at line 83. The orchestrator cannot run.
• profile-ios-sim.sh runs macOS sample; its xctrace line is printed as a suggestion and never executed. No Instruments capture is produced.
• generate-native.sh also targets an iOS simulator binary, so no generic-native platform is separately qualified.
• generate-android.sh runs adb shell rm /data/local/tmp/perf.data and then adb shell simpleperf report -i /data/local/tmp/perf.data, deleting the capture before reporting from it, with the failure suppressed by || true.

So C1 is repair, qualification and then promotion. The reusable work is real; the readiness claim came from reading filenames. Before any five-runtime statement, publish a per-platform status matrix with an actual capture attached to each row.

Two further contracts this needs. A .cpuprofile, a sample tree or an Instruments artifact does not become a Perfetto trace by being moved, so keep original captures and specify conversion and coverage explicitly. And capture identity alone does not make two runs comparable — retain workload, build mode, tool version, device and runtime, warmup count and repetitions with every benchmark result.

26.4 Corrections to §2

• ReaktorWebVitals is a real collector of approximate values. It genuinely observes LCP, CLS with hadRecentInput filtering, INP from first-input and long event entries, long tasks, navigation and resource timing, so the earlier "prototype" status was wrong. The values are approximations of the published metrics, and the earlier correction overstated them: CLS sums shifts across the whole page lifetime where the metric uses the largest qualifying session window, and INP takes an unconditional maximum and folds in a first-input processing interval, where the metric groups interactions, discards the highest per fifty, and measures latency to next paint. Either qualify against the maintained web-vitals implementation and its lifecycle behaviour, or label these as approximations in the report. Test separated shift bursts and a session with many interactions.
• Mobile vitals have a model and no source. There is no JankStats, FrameMetrics, Choreographer, CADisplayLink, os_signpost or Trace.beginSection anywhere in either repository, and reaktor-performance has no androidMain and no iosMain source set at all. JankStats and CADisplayLink bridges are roughly fifty lines each and are the cheapest way to give §6 and §7 a floor. The collector's fixed SlowFrameMs = 16.7 is wrong on any display that is not 60 Hz, so refresh-rate-aware semantics ship with the bridge or the numbers carry an approximation label.
• Telemetry is not merely unexported — it defaults to nothing. TelemetryAdapter's own comment reads "Uses NoopOpenTelemetry by default — zero overhead when no SDK is configured", io.opentelemetry.kotlin:noop is an api dependency, and no exporter is configured in BestBuds. GraphTelemetry produces correct spans from lifecycle, backStack and port events, and they go nowhere. One OTLP exporter changes this, and it unblocks the telemetry/spans and telemetry/metrics workbench subpanes, whose declared blocker is exactly "a Grafana or ClickHouse read adapter".
• The budget checker has no budgets. ReaktorPerformanceBudget, budgetViolations() and requireWithinBudgets() are implemented, and the only budgets constructed anywhere in either repository are inside ReaktorLighthouse.kt. BestBuds defines zero. Principle 3 — metrics require a defined failure threshold — is currently unenforced by its own repository.
• reaktor-performance is used by tests, not by the app. Five call sites: four jvmTest files and one jsMain/BootBridge.kt.

26.5 Benchmarking as a separate discipline

§2 through §25 cover profiling and production telemetry. Benchmarking — repeatable, isolated, comparable measurement of a known workload — is a separate concern with a separate failure mode, and the framework's only serious example of it is the flexbuffer suite.

Two observations that change how it should be positioned:

• Only Android, among every tool in the DevTools comparison, has a first-class benchmarking library (androidx.benchmark, Macrobenchmark and baseline profiles). Xcode has XCTest.measure with baselines. Chrome, React DevTools, Flutter DevTools and Flipper have none. The flexbuffer suite is competitive with androidx.benchmark on rigour; it simply is not a framework service.
• Nothing in that comparison set benchmarks the same source on five runtimes and compares the results. Reaktor already does — JVM, JS, Android, iOS and native, one benchmark, one report. That is a genuinely unusual capability and it is currently a private detail of one module.

26.6 Trace format: adopt Perfetto, do not extend ReaktorFlamegraphFrame

ReaktorProfilerKind already lists Perfetto. Make it the container rather than one of six peers. Android's system trace is Perfetto, Chrome's performance traces export to it, and the DevTools design already asks for SQL over the trace by pointing the Database mode's query workbench at a trace artifact. Emitting Perfetto protos gets all of that for free and makes Reaktor captures readable in ui.perfetto.dev by someone who has never seen this workbench.

ReaktorFlamegraphFrame stays as the normalised summary that budgets are checked against — small, serialisable, diffable in a PR. The full capture is a Perfetto trace referenced by ReaktorProfileCapture. Two artifacts, two jobs: one to gate CI, one to investigate.

26.7 A profile capture is a tooling-layer fact

Every profiler named in §6–§11 is reached by a process the desktop spawns: adb shell simpleperf record, xctrace, async-profiler's agent, wrangler tail. Those are probes, and a profile is a fact on an attachment carrying its rung on the fidelity ladder — L2 for a one-shot capture, L3 for a continuous one, L4 once the in-process tap attributes samples to graph nodes.

This matters for a reason beyond tidiness: a sampled profile taken through a debugger or under an instrumented build is perturbed, and the ladder is where that is recorded rather than assumed away. ReaktorProfileCapture already exists in the model and is the natural carrier.

26.8 Where Reaktor can beat the field

Two capabilities are within reach and are not available in any tool surveyed:

Field telemetry attributed to a graph node. Only two of the reference tools do production field telemetry at all — Xcode via MetricKit and Android via Play Vitals — and both report against a process, a build and a screen. Neither can attribute a regression to a port, an actor or a service boundary, because neither has a typed whole-system graph. Reaktor has the graph, has node.id on every span GraphTelemetry already emits, and has the analytics_events table. Principle 1 is not just a style rule; it is the differentiator.

One benchmark, five runtimes, one report. As above. Nothing else in the category attempts it, and the harness that does it already exists.

Both are blocked on the same two unglamorous things: an exporter behind the noop, and a budget someone is willing to fail a build on.

26.9 Order of work

1. Repair, qualify, then promote flamechart/ into reaktor-performance, parameterised over a target. Adopt where a library exists: async-profiler's Java API (one.profiler.AsyncProfiler, in-process) replaces the JVM script outright, and JFR (jdk.jfr, in the JDK) needs no dependency at all. Perfetto emission uses protobuf-java against the published protos, since no writer library exists. See the build-versus-adopt table.
2. Put a real OTLP exporter behind the noop, using opentelemetry-java on JVM and Android. Its SDK already provides views, aggregation and cardinality limits with overflow counters, so the cardinality work becomes configuration. The experimental io.opentelemetry.kotlin path then only has to carry iOS, JS and Workers. GraphTelemetry's spans are not ready to export — it calls startSpan and addEvent and never calls .end(), and a processor exports completed spans.
3. Define budgets BestBuds is willing to fail a build on. The checker is built and idle.
4. Emit Perfetto for captures; keep ReaktorFlamegraphFrame as the CI summary (§26.6).
5. Bridge mobile frame vitals — JankStats and CADisplayLink, ~50 lines each (§26.4).
6. Make profile capture a probe on the tooling layer, with a rung (§26.7).

Steps 1–3 are the ones that turn this document from a plan into a gate.

Related docs

• Analytics + Telemetry - Use before adding app analytics events, telemetry spans, signal sinks, dashboards, privacy lint, or graph-linked Insights panels.
• Shadow Dev - Use before building production-like test and replay environments.
• DevOps - Use before changing the build/deploy/ops pipeline, the Kotlin platform model, deployment partitioning, gated deploy operations, or the reaktorDesktop Deploy pane; pairs with reaktor-cloud-pane (operator view) and secrets-management.
• Cloud Pane - Use when building the operator view over product infrastructure and tool execution.
• FlexBuffer - Use before changing binary serialization or reaktor-flexbuffer hot paths.
• Server - Use before adding any production backend primitive (resilience, messaging, data patterns) or designing the Spring/Workers server runtime.