Reaktor Server — production backend primitives

What turns reaktor from "a graph runtime" into a production backend: the discipline every real service needs around commands, invariants, durable facts, controlled side effects, reconciliation, and audit — expressed as reaktor-graph constructs (interceptors, facets, passes, actors, the bus) and run on Spring Boot (JVM / k3s) and Cloudflare Workers (edge). The concrete primitives are idempotency, retries, rate limiting, circuit breakers, queueing, batching, caching, state machines, ledgers, the transactional outbox, CQRS, sagas, and event sourcing. The honest framing up front: the service + transport + hosting + state substrate is real and shipping; the resilience + messaging layer is almost entirely net-new — including a declared-but-unused arrow-resilience dependency this doc finally puts to work. And beyond correctness, §9 takes reaktor to high scale and high availability — overload protection, elasticity, sharding, replication, multi-region, zero-downtime deploys, multi-tenancy, coordination, and disaster recovery — where the JVM assembles the primitives from libraries + k3s + GCP, and the Cloudflare edge largely absorbs them.

• Runtimes Spring Boot 4 / WebFlux on k3s (JVM) · Cloudflare Workers + Durable Objects (edge)
• Author once a Service of typed RequestHandlers on the graph; deploy to either runtime
• Cross-cuts as middleware resilience = a ServiceInterceptor stack (Arrow Schedule/CircuitBreaker, resilience4j, Bucket4j); never per-handler code
• Async = reaktor-bus over GCP Pub/Sub + Cloudflare Queues, with transactional outbox + idempotent consumers = effectively-once
• State = actors + ObjectStore / Durable Objects — single-writer, serialized; the substrate for caching, idempotency, sagas, and event-sourced aggregates
• Domain truth = commands + durable facts + projections — handlers validate intent, append facts, derive read models, and reconcile external truth
• The honest invariant exactly-once delivery is impossible (Two Generals / FLP); reaktor targets at-least-once delivery + idempotent processing = effectively-once
• Built for scale load shedding + adaptive concurrency, autoscaling (HPA/KEDA), sharding + hot-key fan-out, read-replicas + read-your-writes, multi-region + residency, zero-downtime canary/blue-green, multi-tenancy + cells, feature flags, and DR (RPO/RTO) — the JVM assembles them, the edge largely absorbs them (§9)

1. Vision

reaktor already models a product as a graph and can stand up a Spring server or a Worker that serves Service handlers. What it lacks is everything between "a request arrived" and "the effect happened durably and exactly the right number of times under load and failure." That gap — resilience, messaging, and data discipline — is what separates a demo from a production backend, and it is what reaktor-server fills.

The design discipline is the same as the rest of the framework: add as little new runtime as possible, and add it as a module, not a kernel change. Concretely, the primitives below are not a new server framework bolted onto reaktor — they are:

• interceptors in the existing ServiceInterceptor chain (timeouts, retries, circuit breakers, rate limits, idempotency, auth),
• facets and passes over the graph (effect classification, outbox wiring, deployment partitioning),
• actor / ObjectStore / Durable Object patterns (caching, dedup stores, saga orchestrators, event-sourced aggregates),
• and reaktor-bus (the messaging substrate over Pub/Sub + CF Queues) for everything async.

Authored once on the graph, a service runs on Spring Boot (heavy, stateful, k3s) or Cloudflare Workers (edge, latency-sensitive) — the resilience and data behavior is the same; only the implementation substrate differs (Arrow/resilience4j on the JVM; Durable Objects/Queues at the edge).

1.1 The production model: commands, facts, invariants, controlled effects

Small apps model the world as row updates plus side effects:

update row
send email
call payment API
change status

Production systems model the same workflow as an auditable chain:

accept command
deduplicate command
enforce invariant
commit durable fact
publish event safely
retry controlled side effects
reconcile external truth
derive current state

That shift is the heart of reaktor-server. A RequestHandler should be an intent boundary, not a generic table patch. Users issue commands (cancelOrder, reserveInventory, capturePayment, grantAccess); the system decides whether the command is legal, records the fact that explains the change, and triggers side effects through an outbox/queue path that can be retried without duplicating the business effect.

The rule is deliberately stronger than "do not write CRUD." CRUD is fine for profiles, settings, and low-risk metadata. It is the wrong model for money, inventory, credits, reservations, entitlements, shipments, notifications, workflows, and any flow where duplicate events, delayed truth, partial failure, correction, audit, or replay matter.

2. Implementation state

Written against the live code in /Users/ovd/dev/reaktor and /Users/ovd/dev/bestbuds. The split between real substrate and missing resilience is stark and worth stating plainly.

Takeaway: the request path, transports, hosting, and state are production-shaped; the reliability and messaging discipline on top of them is the work. reaktor-server is that discipline.

3. Constraints the design must respect

Decisions already made across the kernel, actor, data, auth, telemetry, secrets, and unified-graph docs. The server doc consumes these; it does not re-decide them.

1. Module, not kernel. reaktor-server is a deployment target + module composition, adding interceptors/facets/passes/adapters — never new kernel primitives (graph-kernel-analysis; "no 80 frameworks in a trench coat").
2. Build on the existing local ActorNode; don't replace it. Distribution is ClusterActorRef + k3s StatefulSet + ShardCoordinator + Pub/Sub ordering keys + CF Durable-Object grains, with single-writer via leases + fencing tokens and a consistency-over-availability CAP choice (reaktor-distributed-actors).
3. reaktor-bus is the messaging substrate and a hard dependency — outbox (effectively-once), idempotent consumers, DLQ, claim-check, the correlation/causation envelope. Consume it; do not privately reinvent it.
4. The transactional outbox row is written in the same DB transaction as the domain write. That is the canonical effectively-once pattern; server-issued events must follow it (reaktor-notifications-design).
5. Pub/Sub orderingKey = address is the durable backbone; gRPC is the same-region hot path. Pub/Sub (tens–hundreds of ms) is the durable/cross-region fallback, not the interactive path.
6. Six transports, transport-neutral operation identity, ServiceNode + ServiceInterceptor chain. New endpoints are Service handlers carrying KSerializer + ServiceEndpoint; all cross-cuts attach via interceptors, never inside handler bodies.
7. Data authority and placement follow qualified contracts. The Data Layer uses stable logical data identities, explicit authority/consistency domains and capability-checked bindings. Postgres, Memgraph, ClickHouse, D1/DO/KV, R2 and delivery services are role-aligned choices; no one topology is a framework invariant.

4. The shape — one service, two runtimes, cross-cuts as middleware

A service is authored once as a ServiceNode wrapping a Service of typed RequestHandlers. The production behavior is layered on by the ServiceInterceptor chain (already in the kernel) plus the bus for async and actors/ObjectStore for state. The same authored graph is mounted on Spring (toRouter) or Workers (toHono).

            ┌──────────────── authored once on the graph ───────────────┐
            │  ServiceNode → Service → RequestHandler<In,Out> (×N)        │
            └────────────────────────────┬───────────────────────────────┘
                                          │  mounted on a runtime
        ┌─────────────────────────────────┴─────────────────────────────────┐
   Spring Boot (JVM / k3s)                                   Cloudflare Workers (edge)
   Service.toRouter() → WebFlux                              Service.toHono() → Hono
        │                                                            │
        ▼   ── the ServiceInterceptor stack (SERVER phase) ──        ▼
   [ Auth → RateLimit → Idempotency → Bulkhead → Timeout → CircuitBreaker → Retry → OTel ] → handler
        │ effects                                                    │ effects
        ▼                                                            ▼
   reaktor-bus (publish via OUTBOX in same txn) ──► Pub/Sub / CF Queues ──► idempotent consumer
   actors + ObjectStore (state, cache, ES aggregate)                 Durable Objects (state, cache, dedup, saga)

The interceptor stack is the heart of the design. Each resilience primitive is one interceptor, composed in a fixed, sensible order, and the same Service graph gets the stack on both runtimes — only the implementation of each interceptor differs (Arrow/resilience4j on JVM; Durable-Object-backed at the edge).

// reaktor-server: the resilience stack is just ServiceInterceptors (kernel-native)
service.use(
    AuthInterceptor(authorizer),           // build AuthContext, run LocalAuthorizer (§12)
    RateLimitInterceptor(policy),          // token-bucket / sliding-window  (§5.4)
    IdempotencyInterceptor(dedupStore),    // idempotency-key dedup          (§5.6)
    BulkheadInterceptor(limits),           // concurrency isolation          (§5.5)
    TimeoutInterceptor(deadline),          // never wait forever             (§5.1)
    CircuitBreakerInterceptor(breaker),    // fail fast on a sick dependency (§5.3)
    RetryInterceptor(schedule),            // exp backoff + jitter           (§5.2)
    OtelServiceInterceptor(),              // spans tagged with node.id      (§12)
)

5. Resilience middleware

Each primitive: the best practice, its reaktor mapping (interceptor/facet/actor/DO), the JVM implementation, the edge implementation, and status. The cross-cutting truth — retrying a non-idempotent operation is a bug — ties §5.2 and §5.6 together.

5.1 Timeouts

Best practice (Nygard): never wait forever; an un-timed call is the seed of resource exhaustion and the failure detector a circuit breaker needs. Reaktor: TimeoutInterceptor wrapping the chain. JVM: coroutine withTimeout / resilience4j TimeLimiter + explicit client read/connect timeouts. Edge: Workers have a wall-clock/CPU budget; set explicit fetch timeouts via AbortController. Status: absent.

5.2 Retries

Best practice: retry only transient failures, with exponential backoff + jitter, a max-attempt/elapsed cap, and a global retry budget; pair with a circuit breaker so retries don't pile onto a dying dependency. Reaktor: RetryInterceptor carrying a policy value. JVM — finally use Arrow: Schedule is the gold-standard composable policy —

// Put the declared-but-unused arrow-resilience to work.
val policy = Schedule.exponential<Throwable>(10.milliseconds)
    .doWhile { _, elapsed -> elapsed < 60.seconds }
    .andThen(Schedule.spaced<Throwable>(60.seconds) and Schedule.recurs(100))
    .jittered()                                  // exp → capped-spaced, jittered, as one value
// policy.retry { handler.invoke(request) }      // retry on transient failure

resilience4j Retry (IntervalFunction.ofExponentialRandomBackoff) is the annotation-friendly alternative. Edge: CF Queues retry natively (max_retries default 3, msg.retry({delaySeconds}), up to 24 h → DLQ); for outbound HTTP, backoff+jitter by hand. Status: Arrow declared/unused; current shared wiring is the legacy Meeseeks facade plus socket reconnection. The Work replacement removes Meeseeks and assigns one bounded retry owner per domain, transport, broker or Workflow step.

5.3 Circuit breakers

Best practice (Nygard, Release It!): Closed → Open (fail fast) → Half-Open (probe) state machine, shared across all callers of a dependency; trip on failure-rate over a window (4xx usually doesn't count; slow calls do); needs a timeout (§5.1) as its detector and a graceful fallback. Reaktor: CircuitBreakerInterceptor holding one shared breaker per downstream. JVM — Arrow:

val breaker = CircuitBreaker(                    // NB: CircuitBreaker.of is deprecated (Arrow 1.2)
    openingStrategy = OpeningStrategy.SlidingWindow(windowDuration = 60.seconds, maxFailures = 20),
    resetTimeout = 5.seconds,
    exponentialBackoffFactor = 1.2, maxResetTimeout = 60.seconds,
)
// breaker.protectEither { handler.invoke(request) }  // Either<ExecutionRejected, Out>

or resilience4j CircuitBreaker (COUNT/TIME sliding window, slowCallRateThreshold). Edge: no primitive — hold breaker state in a Durable Object fronting the flaky upstream (count failures in DO storage, flip state, use a DO alarm for the Half-Open probe). The DO gives a shared, consistent breaker across ephemeral isolates — which in-isolate memory can't. Status: absent.

5.4 Rate limiting

Best practice: token bucket (burst-tolerant) vs sliding window (smooth, no boundary-spike); distributed limiting needs shared state (Redis atomic / a single coordinating DO); decide fail-open vs fail-closed when the limiter store is down. Reaktor: RateLimitInterceptor keyed by principal/tenant/route. JVM: ⚠ Arrow has no rate limiter — use resilience4j RateLimiter (note: fixed-window, not token-bucket) or Bucket4j (true token bucket, Redis/Hazelcast-backed for distribution). Edge: route all requests for a subject to one Durable Object by name — single-threaded, transactional storage → race-free buckets without external locks; refill via DO alarm. (Cloudflare's WAF rate-limit is the coarse IP-level complement.) Status: absent.

5.5 Bulkheads

Best practice (Nygard): isolate resources (separate pools / DO partitions) so one saturated dependency can't starve the rest. Reaktor: BulkheadInterceptor; per-tenant/per-resource actors or DOs are bulkheads by construction. JVM: resilience4j Bulkhead (semaphore) / ThreadPoolBulkhead. Edge: per-resource DOs. Status: absent.

5.6 Idempotency

The data idempotency contract owns scoped keys, atomic result persistence, external-effect reconciliation and dedup retention. An interceptor propagates that contract; it cannot turn an arbitrary external call into a database transaction. Shared automatic middleware remains proposed.

6. Messaging & async — reaktor-bus

Durable transport belongs to the Bus design; Cloudflare delivery and multi-step execution belong to Queues and Workflows. Provider limits and ordering are defined there rather than duplicated in this server guide.

6.1 The envelope and the two transports

See the Bus envelope contract.

6.2 Transactional outbox — the dual-write fix

See the durable admission and relay.

6.3 Idempotent consumers, DLQ, backpressure, claim-check

See the Bus delivery contract.

6.4 Resilience composition on the bus

See the Bus delivery contract.

6.5 Effectively-once, not exactly-once

See the Bus delivery contract.

7. Data & state patterns

See the canonical data patterns chapter.

7.1 Caching

See the canonical data patterns chapter.

7.2 Batching

See the canonical data patterns chapter.

7.3 The authority/cache topology

See the canonical data patterns chapter.

7.4 Ledger-style invariants for scarce or conserved things

See the canonical data patterns chapter.

7.5 Reconciliation, repair, and derived state

See the canonical data patterns chapter.

8. Transactional patterns

See the canonical data patterns chapter.

8.1 Command-shaped APIs, state machines, and temporal truth

See the canonical data patterns chapter.

8.2 Saga

See the canonical data patterns chapter.

8.3 Transactional outbox

See the durable admission and relay.

8.4 CQRS

See the canonical data patterns chapter.

8.5 Event sourcing

See the canonical data patterns chapter.

9. Engineering for high scale

The §5–§8 contracts describe how a single service can be made correct under failure. High scale and availability is a second layer — overload, elasticity, partitioning, multi-region, zero-downtime deploys, tenant isolation, coordination, disaster recovery, and the discipline to prove it. The reaktor stance on all of it is one idea: scale behavior is declared in the graph and derived, not hand-wired per service. Every concept below lands on a named reaktor primitive — an existing one extended, or a new one proposed in the same taxonomy: a facet on a node, a pass over the graph, a ServiceInterceptor, a GraphPatch, a DataDescriptor field, a MergePolicy, an ActorNode/Durable-Object pattern, a ClusterActorRef placement policy, a capability. The concept composes with the rest of reaktor instead of bolting a parallel ops stack onto the side.

That gives a clean 80 / 20 split. reaktor encodes the 80% that has semantic meaning in its model — which node sheds load, which key is hot, which DataDescriptor is EU-resident, which contract is evolving, which tenant maps to which shard — as graph metadata the runtime reads. The 20% that is genuinely hyperscale infrastructure — the cross-region consensus engine, the volumetric DDoS scrubber, the pod autoscaler's control loop — is delegated to the JVM/GCP and Cloudflare backends, which become executors of a reaktor-declared intent, never the place the intent lives. The recurring asymmetry holds: the JVM assembles the executor from libraries + k3s + GCP; the Cloudflare edge largely is the executor (implicit autoscale, built-in WAF/DDoS, Durable Objects collapsing partitioning + coordination + per-tenant isolation + single-writer locking, Hyperdrive pooling, D1 Time Travel PITR, Flagship flags).

9.1 Overload protection — load shedding, admission control, adaptive concurrency

Reaktor primitive: a LoadShedInterceptor at the very front of the §5 chain (ahead of even auth, for cheap rejects), with criticality carried on the AuthEnvelope (already part of the bus/auth model) and the adaptive-limit state held in a single-writer ActorNode/ObjectStore counter — so "how loaded is this node" is graph-attributed and reuses the §12 node.id telemetry pipeline rather than a side channel. Best practice (Google SRE, Handling Overload / Cascading Failures): shed at the overloaded resource; meter the constrained resource (in-flight/CPU, never raw QPS); shed lowest-criticality first; the refinement is adaptive concurrency (the limit as a TCP-style congestion window discovered from a latency gradient — Netflix concurrency-limits, Gradient2Limit/VegasLimit); brownout drops optional work within a request first. 80/20: reaktor owns where the interceptor sits, what criticality means (an AuthRequirement-adjacent field), and the binding to §5.2's retry-amplification trap (a 3×-retried shed = ~4× the overload → pair with retry budgets + honored Retry-After); the gradient algorithm itself is the backend library. JVM: Netflix concurrency-limits as the interceptor body + resilience4j Bulkhead (§5.5). Edge: WAF rate-limit rules shed before the Worker runs; in-Worker admission control is a sharded counter DO (one DO's ~1k req/s soft limit makes a single global limiter its own bottleneck). Status: absent.

9.2 Elasticity & autoscaling

Reaktor primitive (proposed): a scaling facet on a ServiceNode/ActorNode declaring its saturation signal (in-flight from §9.1, bus-lag, ObjectStore pressure) and bounds, plus a deploy pass (reaktor-cloud-pane) that compiles that facet into the backend autoscaler config. The graph already knows each node's transport and work-shape, so the scaling signal is derivable from the graph, not hand-tuned YAML. Best practice: scale on the signal that predicts saturation, not CPU-after-the-fact; KEDA scales on queue depth and uniquely scales to zero; keep min-replicas to mask cold starts; beware flapping (stabilization windows) and scaling past a downstream cap (§9.6). 80/20: reaktor declares what signal, what bounds per node (the 80%); HPA/KEDA's control loop is the executor (the 20%), fed by node.id metrics. JVM: the pass emits an HPA custom-metric or KEDA ScaledObject from the facet; counter JVM cold starts with AppCDS / GraalVM-AOT (Spring) or minReplicas. Edge: Workers/DOs auto-scale implicitly — the facet instead drives DO shard count + Queue consumer concurrency + Smart Placement. Status: absent — the k8s manifests in bestbuds/cloud/k3s run fixed replicas.

9.3 Partitioning, sharding & hot keys

Reaktor primitive: the designed ShardCoordinator (256 virtual shards, rendezvous hashing — reaktor-distributed-actors) is the consistent-hash ring; a hot ("celebrity") key is relieved by a GraphPatch that splits one ActorNode into entity:{id}:{n} grain shards behind a supervisor ContainerNode that scatter-gathers — partitioning expressed as a graph transform, not bespoke routing code. Best practice (DDIA Ch. 6): hash/consistent-hash spreads load and limits reshuffle but loses range scans; the hot-partition problem has no general fix beyond salting known-hot keys (fan across N buckets, scatter-gather on read). 80/20: reaktor owns the ring, the fan-out patch, and which keys are declared hot; the backing store's auto-split is the executor. JVM: route via ShardCoordinator; prefer managed sharding (Spanner auto-split, Bigtable row-key design) and salt monotonic keys (timestamps). Edge: Durable Objects are the partition primitive (each named DO = a shard with its own compute + SQLite); the fan-out patch targets entity:{id}:{n} DOs past the ~1k req/s soft-limit; D1 shards app-side across DBs / DO-SQLite. Status: designed (ring), partial (no hot-key patch helper yet).

9.4 Replication & read scaling

Reaktor primitive: read-your-writes is already expressible through SyncCommit's HLC — a session reads at-or-after the HLC of its last write (the same bookmark idea as D1 Sessions) — and DataDescriptor (which already splits authority / projections / caches) gains a readPreference + lag gate. Replica routing is thus a data-model concern reaktor already owns, not a datasource hack. Best practice (DDIA Ch. 5): leader/follower scales reads; the hazard is replication lag → stale read-after-write; fixes are read-your-writes, monotonic reads (sticky replica per session), and quorum R + W > N. 80/20: reaktor encodes consistency intent per DataDescriptor (strong vs read-your-writes vs eventual) + the HLC bookmark; the replica set + lag metric is the backend. JVM: a routing DataSource honoring readPreference; Cloud SQL read replicas gated on the replica_lag SLI, or Spanner bounded-staleness. Edge: D1 read replication + Sessions maps directly onto the HLC bookmark; KV for eventual config; a DO for linearizable read-after-write; Hyperdrive query-caching as a read tier over origin Postgres. Status: absent.

9.5 Multi-region & data residency

Reaktor primitive: this is where reaktor's data model pays off — MergePolicy's CRDTs (PNCounter / ORSet / RGA) are the active-active write-conflict resolution, and the lease + single-writer path (§9.11) is the linearizable alternative; reaktor already makes that consistency choice per DataDescriptor. Residency becomes a DataDescriptor.residency constraint that ClusterActorRef placement and the SyncCommit router honor (a grain + its replicas/backups pinned to an EU node-set); geo-routing is a RouteNode/Selector over the unified system graph. Best practice: active-passive (higher RTO) vs active-active (must resolve cross-region conflicts via LWW/CRDT/consensus); pin data for residency (GDPR); naive multi-master corrupts data. 80/20: reaktor owns conflict semantics (which MergePolicy) and placement/residency (a DataDescriptor field); the cross-region transport + consensus engine is the backend. JVM: GCP global anycast LB; Spanner multi-region (TrueTime) or dual-region for residency as the strong-consistency executor; CRDT descriptors ride async replication. Edge: Cloudflare anycast + Smart Placement; Jurisdictional Restrictions on DOs enforce residency; active-active writes funnel through the per-key DO (single writer). Status: absent (single region asia-south1) — but the conflict/residency contracts remain proposed and must be qualified per runtime. The concrete cross-region data-plane topology — Supabase Postgres, k3s Memgraph + ClickHouse, India + Europe → US + APAC — is in reaktor-db §8.

9.6 Connection management

Reaktor primitive: the edge answer already ships — the Hyperdrive→Postgres adapter (Request.postgres(binding), pooled postgres driver, configurable max, §11); pool saturation becomes a node-level signal on the same OTel node.id pipeline feeding §9.1/§9.2, and JVM pool sizing is a deploy-descriptor field, not ad-hoc config. The bounded pool is just another backpressure boundary alongside the ObjectStore KeyedGate. Best practice: small pools win (HikariCP ≈ (cores × 2) + effective_spindles); front many short-lived clients with a transaction-mode PgBouncer (10k client conns → ~100 server conns); saturation must backpressure (callers block with a timeout), never open unbounded connections; oversized pools are a top self-inflicted DB outage. 80/20: reaktor owns the pool as a declared resource bound + its saturation signal; HikariCP / PgBouncer / Hyperdrive are executors. JVM: Hikari sized per instance × replicas < DB cap; PgBouncer in front of Cloud SQL; never autoscale past the connection budget (§9.2). Edge: Hyperdrive (ships); D1 / DO-SQLite need no external pool (storage co-located with compute). Status: partial — edge ships; the JVM Hikari/PgBouncer discipline (HikariCP has zero usages) is the gap.

9.7 Health, lifecycle & zero-downtime deploys

Reaktor primitive: graceful drain is the ClusterActorRef lease-release + grain-migration the distributed-actors design already specifies (release lease → migrate grain → flip readiness); canary/blue-green is a graph-version GraphPatch that a RouteNode/Selector routes a traffic % to — version routing is native to the unified system graph. Best practice: split the three probes — liveness (cheap, dependency-free, or a shared-dependency blip restarts every pod → cascading-restart amplification), readiness, startup; drain on SIGTERM with a preStop bridge; escalate rolling → blue-green → canary (ramp % + auto-rollback); warm the JVM. 80/20: reaktor owns the drain choreography + the version-routing patch; k8s probes/PDB and CF Gradual Deployments are executors. JVM: Spring Actuator probe groups + server.shutdown=graceful + preStop sleep + maxUnavailable=0 + PodDisruptionBudgets; canary via Argo Rollouts. Edge: atomic wrangler deploy + Gradual Deployments (% split, one-step rollback); no probe/drain model, but DO code changes must tolerate in-flight migration (idempotent handlers; schema via §9.8). Status: partial — boots, but no probes/drain/canary wired.

9.8 API contract & evolution

Reaktor primitive: the contract is the ContractId-typed port system — provider/consumer ports already enforce structural compatibility, so expand-contract becomes a GraphPatch sequence (add provider port → dual-serve → remove old), and the bus Envelope already carries schemaVersion. problem+json is an error-shaping ServiceInterceptor (CLIENT/SERVER phase); pagination a RequestHandler convention. Best practice: version explicitly, keep changes additive, migrate via Parallel Change / expand-contract so versions coexist across a deploy; prefer keyset over offset pagination (offset is O(n)-deep and skips/dupes on shifting data); ship RFC 9457 application/problem+json (obsoletes 7807) + validation + compression. 80/20: reaktor owns contract identity + the migration patch + the error shape; Flyway / D1-migrations are the schema executors. JVM: Spring ProblemDetail, Bean Validation, keyset cursors over Spring Data's offset Pageable, Flyway expand-contract gated per app version. Edge: RFC 9457 from the Worker; keyset over D1 / DO-SQLite; D1 migrations so a Worker mid-Gradual-Deployment (§9.7) works against either schema. Status: absent — but ContractId already gives the compatibility check.

9.9 Multi-tenancy, quotas & cells

Reaktor primitive: tenantId already flows through AuthContext and the bus Envelope, so quotas are a RateLimitInterceptor keyed by AuthContext.tenantId and metering is OTel tenant attribution (a bounded label, honoring the §12 cardinality contract). Structurally, a cell is a partition of the unified system graph placed on a node-set, shuffle-sharding is a ClusterActorRef assignment policy (each tenant → a random subset of the 256 shards), and tenant→cell routing is a Selector/RouteNode. Best practice: silo / pool / bridge isolation; defend the noisy neighbor with quotas + shuffle sharding (blast radius ≈ one virtual shard — AWS Builders' Library) + cell-based architecture (contained failure — AWS Well-Architected); meter by aggregation (hot counters need §9.3 salting). 80/20: reaktor owns tenant identity, quota keying, cell partitioning, shard assignment; namespaces / Analytics Engine are executors. JVM: RLS / Hibernate tenant_id scoping, resilience4j limiters keyed by tenant, cells as namespaces behind a tenant→cell Selector. Edge: a DO-per-tenant is a cheap silo; quotas in the DO; cells as DO/Worker namespaces; meter via Workers Analytics Engine. Status: absent — identity flows, the quota/cell/metering layer does not.

9.10 Progressive delivery & feature flags

Reaktor primitive (proposed): a flag is a capability (reaktor-capability) plus a node-activation facet evaluated by a pass (flip a node on/off in the graph without redeploy); a kill switch is a SafetyClass gate (reaktor already gates risky operations by safety class); a % rollout is a Selector over AuthContext.identityId with consistent hashing (same user → same variant). "Decouple deploy from release" thus maps onto capability-gating + graph activation, not an external config blob. Best practice (Fowler): toggles (release / ops kill-switch / experiment / permission); layer canary-% / dark-launch / rings; a kill switch is non-negotiable; prune flag debt; the flag store is a hot-path dependency (cache + safe default). 80/20: reaktor owns what a flag gates (a capability/node) + rollout cohorting; the flag store/SDK is the executor. JVM: an OpenFeature provider behind the capability check. Edge: Cloudflare Flagship (native, OpenFeature, evaluated in-Worker with no outbound call, KV+DO-backed, consistent-hash % rollout) is the edge store. Status: absent — adopt OpenFeature as the shared API so JVM + Flagship share flag definitions.

9.11 Distributed coordination

Reaktor primitive: the distributed-actors design already specifies single-writer via leases + monotonic fencing tokens — reaktor's answer to a lock is "route the mutation through the entity's single-writer grain," and where an external resource is touched the grain issues a fencing token the resource enforces (WHERE token > current). Best practice (Kleppmann, How to do distributed locking): a lock is safe only with fencing tokens — a process paused (GC, page fault) past lease-expiry otherwise wakes and corrupts data (why Redlock is unsafe for correctness); prefer single-writer; use HLCs for causal order over wall clocks. 80/20: reaktor owns the single-writer model + token issuance/enforcement; etcd / Spanner is the optional consensus backend — often unneeded, since a transactional read-modify-write suffices. JVM: the fencing token as an optimistic-lock version; a Spanner/Postgres txn often removes the external lock entirely. Edge: Durable Objects make most locks unnecessary — one single-threaded instance per id serializes all mutations by construction; a DO issues fencing tokens from its SQLite; DO alarms handle lease timeouts. Status: designed (JVM leases/fencing), intrinsic (edge DO model).

9.12 Disaster recovery & data lifecycle

Reaktor primitive: SyncCommit is already a content-addressed, HLC-ordered log — i.e. a restorable history — and the DataDescriptor authority-vs-projection split defines backup scope by construction: authority stores need PITR; projections/caches are rebuildable (the doctrine "caches are materialized views, never accidental second truths" is the RTO strategy). So RPO/RTO targets attach to a DataDescriptor, PITR is SyncCommit replay (or D1 Time Travel), and GDPR erasure is a GraphPatch + crypto-shred key in reaktor-auth. Best practice: set RPO/RTO per workload; backup + PITR; test restores (an untested backup is not a backup); lifecycle TTL/tiering; compliant deletion (crypto-shred immutable backups); at scale RTO is bounded by restore time → replica promotion beats restore. 80/20: reaktor owns what must be backed up vs rebuilt + the erasure transform + restore targets; the WAL-archiver / Time-Travel engine is the executor. JVM: Cloud SQL PITR + promotable cross-region replicas; GCS lifecycle; an erasure workflow across stores. Edge: D1 Time Travel PITR; R2 lifecycle + DO-SQLite export target. Status: absent — but SyncCommit + the authority/projection split already frame it.

9.13 Security at scale

Reaktor primitive: workload identity is a service principal per node/grain — reaktor-auth already models appId + service principals (client_credentials), which is exactly a SPIFFE-shaped intrinsic identity; mTLS is the binding on the PEER ServiceTransport, and authorization stays the existing capability + AuthRequirement + LocalAuthorizer check at every node. Service-to-service security thus extends reaktor-auth rather than wrapping it in a separate mesh. Best practice: edge DDoS / WAF / bot; service mTLS with auto-rotated workload identity (SPIFFE/SPIRE: attested spiffe:// ID + short-lived X.509-SVIDs); parameterize queries; classify PII (sprawl defeats erasure, §9.12); rotate secrets; an absorbed-but-expensive DDoS is a financial DoS (→ §9.14). 80/20: reaktor owns identity-per-node + capability authz + which transport is mTLS; the cert-rotation daemon and volumetric DDoS scrubber are executors. JVM: SPIRE issues the SVID for each node's service principal; Cloud Armor WAF/DDoS; Secret Manager rotation — the §12 model, fleet-scaled. Edge: CF WAF + always-on DDoS + Bot Management front every Worker; Service Bindings = network-less, unexposed worker calls; Workers Secrets. Status: minimal — the identity model exists; mesh/WAF/SPIFFE wiring does not.

9.14 Capacity, cost & chaos

Reaktor primitive: reaktor has a native chaos substrate the generic stacks lack — the shadow dev environment (capture/replay) over deterministic actors: a fault-injection ServiceInterceptor perturbs a recorded trace and deterministic replay proves the §5 primitives engage (did the breaker open? did load-shedding shed?) — repeatably, not by randomly killing prod pods. Capacity is derived from the graph (a pass sums each node's per-request cost × transport), and cost guardrails reuse OTel tenant/node attribution → ReaktorPerformanceBudgets (already a CI gate). Best practice: load / spike / soak test (k6 — soak catches slow leaks / GC death-spirals); capacity-plan from demand × per-request cost (SRE Non-Abstract Large System Design) on the binding constraint (often connections, §9.6, not CPU); FinOps guardrails (an autoscale or retry storm shouldn't blow the bill); chaos with blast-radius limits + a kill switch. 80/20: reaktor owns deterministic fault injection + graph-derived capacity + budget gates; k6 / Chaos-Mesh are executors. JVM: k6 soak vs SLO p99; GCP Budgets + per-tenant labels → cost analytics in the single OLAP (ClickHouse); Chaos Mesh scoped, tied to the §5 interceptors. Edge: k6 vs DO/Queue limits; Workers Analytics + the WAF rate-limit layer capping the cost-attack surface. Status: absent — but shadow-replay is the unique enabler.

10. Running on Spring Boot (JVM / k3s)

The heavy, stateful runtime. Today: Spring Boot 4 / WebFlux (Netty), Service.toRouter() mounts handlers, 1 replica, stateless, in-memory graph at boot, OCI image → k3s (asia-south1).

What reaktor-server adds on the JVM:

• The resilience stack (§5) backed by Arrow Schedule/CircuitBreaker + resilience4j (RateLimiter/Bulkhead/TimeLimiter) + Bucket4j (token-bucket) — wired as ServiceInterceptors.
• Auth enforcement (the gap, §12): a ReaktorSecurityWebFilter that builds AuthContext from the bearer token (local JWKS verify), runs LocalAuthorizer, plus real SessionStore/RefreshTokenStore, a /.well-known/jwks.json endpoint, client_credentials for service accounts.
• Durable work (proposed) through the Work architecture: an authoritative server outbox/WorkStore, fenced bounded attempts and explicit provider handoffs. Selected profiles are Spring-managed Quartz with persistent JDBC/cluster scheduling and queues/GCP Pub/Sub, and Cloudflare Workflows/Queues/GCP Pub/Sub. Quartz supplies wakeups, brokers own delivery retry and Workflows owns step history. Current Meeseeks wrappers establish none of these replacement guarantees.
• DB/query observability: OtelServiceInterceptor, OtelGraphDb (Cypher), OtelPgInterceptor (parameterised SQL only), pool/pg_stat_statements scrapers.
• The clustered stateful runtime (the big infra gap, reaktor-distributed-actors): k3s StatefulSet + headless Service for stable pod identity, k8s Endpoints watch + phi-accrual failure detector for membership, a lease-protected ShardCoordinator (256 virtual shards, rendezvous hashing), ClusterActorRef resolving Local | Pod | Edge | Inactive, gRPC same-region transport with Pub/Sub (orderingKey) durable fallback, single-writer via leases + monotonic fencing tokens, readiness/preStop drain.

11. Running on Cloudflare Workers (edge)

The latency-sensitive runtime. Today: Service.toHono() mounts handlers; Durable Objects, KV, D1, R2, and Hyperdrive→Postgres (Request.postgres(binding) { … } over a pooled postgres-driver client with a configurable max) adapters are implemented; Cloudflare Queues is not (the QUEUE transport is an enum only).

What reaktor-server adds at the edge — almost everything maps to Durable Objects (single-instance-per-id, transactional + SQLite storage, in-memory state, alarms):

• Idempotency dedup authority (DO keyed by idempotency key) / D1 UNIQUE — not KV.
• Rate limiting — one DO per subject, single-threaded buckets, alarm refill.
• Circuit breaker — shared breaker state in a DO fronting a flaky upstream.
• Saga orchestrator / event-sourced aggregate / hot cache — all DO patterns (§7, §8).
• Async — the CF Queues adapter (the QUEUE transport): batches, max_retries, dead_letter_queue, per-message delaySeconds; the outbox relay is a Cron Worker / DO alarm over D1.
• Auth — a WorkerJwtVerifier (crypto.subtle + KV-cached JWKS).
• Secrets — pushed per-worker at deploy via wrangler secret bulk (Workers can't reach GCP metadata), read via requireSecret(name); most workers get zero secrets.

12. Observability, auth, and secrets

• Observability — OTel is the only wire format; every span/metric/log carries node.id + graph.id (graph attribution is sacred), instrumented outside the node via the GraphTelemetry/interceptor pattern. node.id is trace/log-only, never a metric label (cardinality contract). SLOs drive multi-window burn-rate paging; ReaktorPerformanceBudgets.requireWithinBudgets() gates CI. Operator/agent actions are spans tagged agent vs devtools.
• Auth — every call enters with AuthContext (principal/identityId/appId/tenantId/scopes/roles/delegation); protected capabilities declare an AuthRequirement checked by the multiplatform LocalAuthorizer. PrincipalKind.AGENT is first-class but excluded unless granted. Tokens: short-lived Reaktor JWT (ES256 via JWKS — workers verify locally), rotating refresh tokens (reuse detection), rak_ PATs (exchange-only, audience-bound). Background/delivery work runs as service principals.
• Secrets — GCP Secret Manager ({env}-{domain}-{name}); local gcloud ADC, k3s node-SA ADC + External Secrets Operator, CI GitHub OIDC → Workload Identity Federation; delivered by Dagger functions; least privilege (most workers: zero secrets); per-tenant payload encryption via KMS for hosted notifications.

13. The reaktor-server module

A module composition + deployment story, owning no business logic.

Adds: the ServiceInterceptor resilience stack (Timeout/Retry/CircuitBreaker/RateLimit/Bulkhead/Idempotency); reaktor-bus (envelope, outbox + relay, idempotent consumer, DLQ, claim-check) over Pub/Sub + a new CF Queues adapter; effect-classification + outbox-wiring facets/passes; saga/CQRS/event-sourcing patterns over actors/DOs; server auth middleware (Spring filter + worker JWT verifier + JWKS + sessions + service accounts); DB/query OTel interceptors; the clustered-JVM runtime (ClusterActorRef, StatefulSet, ShardCoordinator); and the §9 high-scale primitives (load-shed interceptor, scaling facet → deploy pass, hot-key GraphPatch, DataDescriptor.readPreference/residency + MergePolicy CRDTs, capability-gated feature flags, SyncCommit-based DR + erasure patch, shadow-replay chaos) — each declared in the graph, with the platform as executor.

Consumes (never reinvents): the kernel + ServiceNode/interceptor chain, ActorNode/ObjectStore, reaktor-db (authority/cache/DataDescriptor), reaktor-auth kernel, reaktor-google Pub/Sub, reaktor-cloudflare (DO/KV/D1/R2/Queues), reaktor-work (qualified scheduling and durable handoff profiles; proposed), the OTel/telemetry pipeline, arrow-resilience + resilience4j + Bucket4j.

Must not: become a parallel mutation path, a second graph runtime, a kernel change, or a private reimplementation of the bus/db/auth kernels.

14. Phased plan

P0 + P1 are the unlock: a resilience stack and a real bus turn the existing service substrate into a production backend. P7–P10 are the scale hardening — sequence them once a single node is correct (P0–P6); coordination + security-at-scale (§9.11, §9.13) fold into P5 (clustered leases + fencing) and P3/§12 (auth + mTLS) rather than standing alone, and most of P7–P10 is JVM work since the edge absorbs much of it.

15. Open decisions

• Arrow vs resilience4j vs Bucket4j per primitive. Recommendation: Arrow Schedule (retry) + CircuitBreaker (they compose with typed errors and read well); resilience4j for Bulkhead/TimeLimiter; Bucket4j for token-bucket rate limiting (Arrow has none; resilience4j's is fixed-window). Confirm the JVM matrix before P0.
• Saga engine. Arrow Saga (in-process) for single-service rollback vs a durable orchestrator (DO / Cloudflare Workflows / Temporal) for long-running cross-service sagas. Likely both, by saga lifetime.
• Outbox relay. Polling (simple) vs CDC/WAL (Debezium/Datastream, efficient, heavier) on the JVM; D1 has no WAL, so edge is polling (Cron/DO alarm).
• Ledger adoption boundary. Use ledger-style modelling for scarce, valuable, reservable, consumable, refundable, externally reconciled, or multi-service-owned resources; keep low-risk CRUD on simpler versioned updates.
• Repair authority. Decide which support/ops roles can post correction facts, how approvals are represented, and which reason-code catalogs are shared across products.
• Exactly-once stance. Adopt effectively-once everywhere; reserve Pub/Sub native exactly-once (pull-only, single-region, ~1k msg/s) for the few flows that justify the throughput cap.
• CF Queues vs Pub/Sub split. Edge-native flows on Queues; cross-region / ordering-key / massive-fan-out on Pub/Sub. One routing table, two adapters.
• Clustered runtime timing. The biggest infra lift (~13–14 eng-weeks per distributed-actors); sequence it after the single-node resilience + bus are proven.
• Overload signal & limiter (§9.1). Adaptive concurrency (Netflix gradient) vs static resilience4j bulkheads; meter on in-flight/CPU, never QPS. Confirm before P7.
• Autoscaling trigger (§9.2). HPA-on-custom-metric vs KEDA-on-bus-backlog (with scale-to-zero for async consumers) — bound by the hard rule that pods never exceed the DB connection budget (§9.6).
• Sharding key & hot-key policy (§9.3). Which entities get pre-emptive salt/fan-out, the salt count, and how it reconciles with ShardCoordinator's 256 virtual shards.
• Feature-flag provider (§9.10). Adopt OpenFeature as the common API — Flagship at the edge, an OpenFeature provider on the JVM — so flag definitions are shared, not forked per runtime.
• Cell granularity & tenancy model (§9.9). Pool-by-default with a silo (dedicated DO / Deployment + DB) for premium tenants; cell size to bound blast radius; whether to introduce cells at all before tenant count warrants it.
• DR targets per store (§9.12). Explicit RPO/RTO for Postgres / Memgraph / D1 / DO state, and whether residency (§9.5) forbids the cross-region DR copy.

16. References

Verified June 2026. The load-bearing impossibility result and all external API names were checked against primary docs.

Books

• Designing Data-Intensive Applications — Martin Kleppmann — O'Reilly · dataintensive.net — the systems-thinking foundation. Chapter map (1st-ed numbers): Ch. 8 distributed-systems trouble → retries/timeouts/why exactly-once-delivery is impossible; Ch. 9 consistency & consensus → sagas-vs-2PC; Ch. 7 transactions → idempotent receivers + outbox atomicity; Ch. 11 stream processing → event sourcing, queueing, batching, backpressure; Ch. 12 derived data → end-to-end idempotency, outbox/CDC.
• Building Microservices, 2e — Sam Newman — O'Reilly — decomposition, inter-service comms, the operational realities behind queueing/sagas/resilience.
• Enterprise Integration Patterns — Hohpe & Woolf — enterpriseintegrationpatterns.com — the messaging-pattern bible: Idempotent Receiver, Dead Letter Channel, Competing Consumers, Claim Check, Guaranteed Delivery.
• Release It!, 2e — Michael Nygard — Pragmatic — origin of the Circuit Breaker; also Bulkhead, Timeout, Steady State.

Pattern catalogs & docs

• microservices.io — Chris Richardson — Saga · CQRS · Transactional Outbox · Idempotent Consumer · Transaction Log Tailing · Circuit Breaker.
• Arrow Resilience — Retry & Repeat (Schedule) · Circuit Breaker · Saga · index. (Arrow's toolkit is exactly these three — no built-in rate limiter; CircuitBreaker.of deprecated in 1.2; Saga is in-process, not a distributed durable engine.)
• resilience4j — docs — CircuitBreaker, RateLimiter (fixed-window, not token-bucket), Retry (ofExponentialRandomBackoff), Bulkhead/ThreadPoolBulkhead, TimeLimiter. Bucket4j — bucket4j.com — true token-bucket, distributable.
• Cloudflare — Queues (batching/retries/DLQ) · Durable Objects (single-writer, transactional storage, alarms) · D1 · KV (eventually consistent) · Workflows (durable execution).
• GCP Pub/Sub — Subscriber types · Ordering keys · Exactly-once delivery.
• Impossibility result — Tyler Treat, "You Cannot Have Exactly-Once Delivery"; the Two Generals / FLP basis for effectively-once.

High scale & operations (§9)

• Google SRE Book / Workbook — sre.google/books — Handling Overload (ch. 21) and Addressing Cascading Failures (ch. 22) for load shedding/criticality; the Non-Abstract Large System Design workbook chapter for capacity planning.
• DDIA — Replication & Partitioning — Kleppmann — Ch. 5 / Ch. 6 — leader/follower, replication lag, read-your-writes, quorums; consistent hashing, the hot-key/celebrity problem, salting.
• AWS Well-Architected — Reliability Pillar · Reducing the Scope of Impact with Cell-Based Architecture.
• Amazon Builders' Library — Workload isolation using shuffle-sharding · Timeouts, retries, and backoff with jitter.
• Martin Fowler (bliki) — ParallelChange / expand-contract · BlueGreenDeployment · CanaryRelease · FeatureToggles.
• Distributed locking — Kleppmann, "How to do distributed locking" — the fencing-token argument and the Redlock critique.
• Adaptive concurrency — Netflix, "Performance Under Load" · Netflix/concurrency-limits (Gradient2Limit/VegasLimit).
• RFC 9457 — Problem Details for HTTP APIs — rfc-editor.org — the current application/problem+json standard; obsoletes RFC 7807.
• SPIFFE / SPIRE — spiffe.io — workload identity (spiffe:// IDs, X.509-SVIDs) for auto-rotated mTLS.
• KEDA — keda.sh — event-driven autoscaling + scale-to-zero (which HPA lacks); GKE scale-to-zero with KEDA.
• HikariCP — About Pool Sizing — wiki — the (cores × 2) + spindles formula; "small pools win."
• Cloudflare — Hyperdrive (edge Postgres pooling) · Smart Placement · WAF rate-limiting rules · Gradual Deployments · Flagship (OpenFeature flags) · D1 Time Travel.
• OpenFeature — openfeature.dev — the CNCF feature-flag API standard (the shared JVM + Flagship surface).
• Grafana k6 — k6 docs — load / spike / soak testing.

See also

• Reaktor Actor Model · Reaktor Distributed Actors — the state + distribution substrate.
• Reaktor Notifications Design — the first consumer of the bus/outbox/idempotency story.
• Shadow Dev Environment — capture/replay that depends on idempotent effects.
• Performance and Observability · Reaktor Analytics and Telemetry — the OTel layer.
• Auth Architecture · Secrets Management — the security layer the server enforces.
• Reaktor Unified System Graph · Reaktor Cloud Pane — deployment + the cloud control plane.
• Reaktor Visual Language — where saga/workflow graphs are authored.

Related docs

• Service Layer - Use before adding service APIs, splitting reaktor-service out of reaktor-graph, wiring handlers to Spring/Workers, or making modules graph-projectable without hard graph dependencies.
• Reaktor Bus - Use before designing durable async events, jobs, fan-out, queues, Pub/Sub, outbox, DLQ, replay, or graph-visible delivery.
• Data Layer - Use before choosing a store, designing offline-first sync, or deciding authority vs cache vs projection for a datum; the centerpiece is the tier 1↔2/3 server-authoritative sync engine.
• Multitenancy - Use before designing tenant isolation, adding a tenant-scoped store, wiring the tid claim, or building per-tenant quotas/residency/lifecycle; pairs with reaktor-server (high-scale tenancy) and auth-analysis (the membership model).
• Auth Architecture - Use before changing reaktor-auth or request authorization boundaries.
• Core - Use before touching the platform boundary: registering a Feature slot, writing a platform Adapter, or branching on a device/network capability (which must be a normalized tier, never a raw model string).