Reaktor Distributed Actors

The local ActorNode in reaktor-graph-runtime already has the hard part — a single-threaded mailbox, supervision directives, an exclusive per-actor ObjectStore, and ask/tell. This document is the plan to make it behave like Erlang/OTP across a global fleet: Spring Boot pods on GCP CE k3s, GCP Pub/Sub for the durable wire, Cloudflare Durable Objects as native edge grains — one location-transparent ActorRef over all of it.

• Builds on — reaktor-graph reaktor-google reaktor-cloudflare reaktor-flexbuffer

• Substrate GCP CE k3s · Pub/Sub (ordering keys) · Cloudflare DO · Memgraph

• Model Orleans virtual actors + OTP supervision

• Status Design

• Local ActorNode — real — mailbox, supervision, ObjectStore, ask/tell

• Pub/Sub ordering keys — real — per-actor order ✓

• CF Durable Objects — real — PartyKit + raw bindings

• FlexCoder serde — real — ready for wire

• Cluster membership — missing

• Sharding / directory — missing

• Remote transport — missing (PUBSUB transport is enum-only)

• app-server replicas today — 1 (no headless svc)

1. Executive summary

Erlang's reputation — millions of cheap processes, supervision trees, let-it-crash, telco-grade uptime — comes from two things the BEAM gives for free: process isolation and location-transparent distribution. Reaktor already has the first (each ActorNode is a coroutine pinned to limitedParallelism(1) with a private Channel mailbox and supervision directives). It does not have the second: ActorAddress is graphId/key, send() only reaches a local channel, and there is no cluster, registry, or remote wire.

The proposal: keep the ActorNode exactly as-is for in-process behaviour, and add a thin cluster layer that makes ActorRef location-transparent across the real infrastructure. Three moves:

• Membership — model each region as a k3s StatefulSet of actor-hosting pods with a headless Service (the exact pattern the Memgraph StatefulSet already uses). Pods discover each other through the k8s Endpoints API; a phi-accrual failure detector marks dead pods.
• Sharding — hash ActorKey.name into one of N virtual shards; a ShardCoordinator assigns shards to live pods and rebalances on membership change. This is the Orleans/Akka-Cluster-Sharding model — the actor is a virtual (Orleans grain): it activates on its shard's owner on first message, passivates when idle, and migrates when ownership moves.
• Transport — local sends stay zero-copy coroutine channel sends. Remote sends pick a wire by SLA: gRPC for same-region hot paths, GCP Pub/Sub with the actor address as the ordering key for durable / cross-region / at-least-once delivery (the ordering key gives per-actor message order — the single most important primitive, and it already exists in reaktor-google). At the edge, Cloudflare Durable Objects ARE Orleans grains natively (single-instance, globally routed, persistent storage) — they join the same ActorRef abstraction as a second runtime.

2. What exists today

From reaktor-graph-runtime/src/commonMain/.../core/node/ActorNode.kt — 421 lines of real, working code:

3. The Erlang/OTP bar — what "used like Erlang" means

To be "used like Erlang" across servers, the system must provide, at minimum:

Location transparency

A sender holds an ActorRef and calls send / ask. Whether the actor is in-process, on another pod, in another region, or a CF Durable Object at the edge is invisible.

Let-it-crash

Actors don't defensively catch. They crash; a supervisor restarts them with clean state. At cluster scale, a pod crashing is "a process crashing" — its actors restart elsewhere.

Supervision trees

Failures escalate up a tree. Restart intensity limits (e.g. 5 restarts / 60 s) prevent crash loops; exceeding escalates to the parent.

Links & monitors

An actor can watch another and receive a Terminated signal when it (or its hosting node) dies — across the network.

Node membership

The cluster knows which nodes are alive (EPMD + net_kernel in Erlang). New nodes join; dead nodes are detected and their work reassigned.

Cheap, numerous, addressable

Millions of potential actors, only the active ones resident in memory (virtual actors), each individually addressable by stable identity.

Back-pressure

A slow consumer suspends its producers rather than melting down. Bounded mailboxes + coroutine suspension already do this locally; it must hold across the wire too.

Hot failover

A node going down (deploy, crash, zone outage) does not lose actors — they re-activate elsewhere from persisted state. Target: sub-second for hot actors.

4. Erlang/OTP ↔ Reaktor mapping

What the BEAM supplies vs what Reaktor supplies on this substrate:

5. Substrate reality check

Grounded in a fresh code survey — what's real to build on vs what the spec/enums only promise:

6. Two runtimes, one ActorRef

There are two natural actor runtimes. The design unifies them behind the existing ActorRef<M> interface so callers never branch on location.

core JVM actors on k3s

Heavy, stateful, compute-bound actors: knowledge-graph workers, ranking, aggregation, ML, the ChatRoomActor server authority. Run as ActorNode on Spring Boot pods, sharded across the StatefulSet. Reaktor owns membership, sharding, registry, handoff.

edge Cloudflare Durable Objects

Latency-sensitive, user-facing, naturally-partitioned actors: chat-room fan-out, presence, typing indicators, per-user session. Run as Durable Objects (already working via PartyKitServer). CF supplies single-instance guarantee, global routing (idFromName), storage, and alarms — Orleans grains for free.

The unifying reference

// commonMain — extend the existing ActorRef with a cluster-aware impl.
class ClusterActorRef<M : Any>(
  override val key: ActorKey<M>,
  private val cluster: ActorCluster,
) : ActorRef<M> {
  override val address: ClusterAddress = cluster.addressOf(key)

  override suspend fun send(message: M) {
    when (val placement = cluster.resolve(key)) {
      is Placement.Local   -> placement.node.send(message)             // zero-copy channel send
      is Placement.Pod     -> cluster.transport.tell(placement.pod, key, message)
      is Placement.Edge    -> cluster.edge.tell(placement.doId, key, message)
      is Placement.Inactive-> cluster.activateThenSend(key, message)         // virtual-actor activation
    }
  }

  override suspend fun<R> ask(timeout: Duration, build: (Reply<R>) -> M): R =
    cluster.ask(key, timeout, build)   // correlation id flows over the wire; reply routed back
}

The local fast path is preserved exactly: if the actor is on this pod, send is the same coroutine channel send it is today — no serialization, no network. Everything else is the cluster layer's job.

7. Cluster membership (the EPMD / net_kernel replacement)

7.1 Stable identities via StatefulSet

The actor-hosting servers become a Kubernetes StatefulSet (not a Deployment) with a headless Service — the exact pattern the Memgraph StatefulSet already uses in cloud/k3s/manifests/memgraph.yaml. This gives each pod a stable identity and DNS name:

reaktor-actor-0.reaktor-actor.bestbuds.svc.cluster.local
reaktor-actor-1.reaktor-actor.bestbuds.svc.cluster.local
reaktor-actor-2.reaktor-actor.bestbuds.svc.cluster.local

Stable ordinals matter: shard assignment and gRPC dialling need addressable, predictable peers — not the random pod IPs a Deployment gives.

7.2 Discovery + failure detection

• Join: a pod starts, registers itself by watching the headless Service's Endpoints via the k8s API (k3s-native, no extra infra). It learns the full live set.
• Heartbeat: pods exchange lightweight heartbeats over gRPC at ~1 s. A phi-accrual failure detector (the Akka/Cassandra approach) converts heartbeat history into a suspicion level, so a slow pod isn't instantly declared dead.
• Leave: graceful shutdown (SIGTERM on rolling deploy) → the pod passivates its actors and announces departure so the coordinator rebalances before the pod dies, avoiding a detection gap.
• Down: failure detector crosses threshold → coordinator marks the pod down → its shards reassign.

8. Addressing & the grain directory

8.1 Cluster address

data class ClusterAddress(
  val scope: Scope,        // Core | Edge
  val region: String,      // "asia-south1" | "us-central1" | "edge"
  val shard: Int,          // 0..255  (hash of key.name)
  val key: String,         // actor name, e.g. "chats/room-42"
) {
  override fun toString() = "reaktor://$region/$scope/$shard/$key"
}

8.2 Resolution

Resolving an ActorKey to a placement is the heart of location transparency:

1. Hash — shard = consistentHash(key.name) % SHARD_COUNT (default 256 virtual shards). Reuses the same stable-hash discipline already in stableActorUuid.
2. Lookup owner — ask the ShardCoordinator which pod owns that shard. Cached locally with invalidation on rebalance events.
3. Local or remote — if owner == self and the actor is active → Placement.Local. If owner is another pod → Placement.Pod. If the key is in the Edge scope → Placement.Edge (CF DO via idFromName(key)). If owner == self but actor is passivated → Placement.Inactive (activate).

8.3 The directory

Two layers, matching the spec's "grain directory" idea but on real infra:

• Shard → pod map: owned by the ShardCoordinator (a cluster-singleton actor), replicated to all pods via Pub/Sub broadcast on every rebalance. Small (256 entries), changes rarely.
• Edge directory: Cloudflare's own routing. idFromName("chats/room-42") deterministically maps a name to a DO — CF is the directory for edge actors. Optionally a CF DO acts as the global region directory ("which region owns this user?"), matching spec §13.2's "Discovery Plane (namespace Durable Object)".

9. Sharding & the coordinator

ShardCoordinator (cluster singleton, lease-protected)
│ owns: shard → pod (256 entries)
│
├─────────────────────────┬─────────────────────────┐
▼                         ▼                         ▼
reaktor-actor-0           reaktor-actor-1           reaktor-actor-2
shards 0–85               shards 86–170             shards 171–255
┌───────────────┐         ┌───────────────┐         ┌───────────────┐
│ ActorNode     │         │ ActorNode     │         │ ActorNode     │
│ chats/r-42    │         │ rank/user-7   │         │ kg/domain-3   │
│ ActorNode     │         │ ActorNode     │         │ ActorNode     │
│ chats/r-9     │         │ rank/user-2   │         │ kg/domain-8   │
└───────────────┘         └───────────────┘         └───────────────┘

pod-2 dies
  ──▶ coordinator detects (phi-accrual)
  ──▶ reassign shards 171–255
  ──▶ split across pod-0, pod-1 (consistent hashing: minimal movement)
  ──▶ affected actors REHYDRATE from ObjectStore on new owners
  ──▶ in-flight messages buffered (Pub/Sub retains) until reactivation

9.1 The coordinator

• What it is: a single actor (yes — an ActorNode) that owns the authoritative shard→pod table. Singleton across the cluster, protected by a lease (k8s Lease object or a CF DO) so exactly one exists even during a partition.
• Rebalancing: on join/leave/down it recomputes the assignment using rendezvous (HRW) hashing — each shard goes to the live pod with the highest hash(shard, pod) score. Minimal movement: only shards whose top-scoring pod changed move.
• Handoff protocol: (1) coordinator tells old owner "stop shard S"; (2) old owner passivates S's actors (snapshot to ObjectStore), acks; (3) coordinator assigns S to new owner and broadcasts the new map; (4) buffered messages (held in Pub/Sub, not lost) flow to the new owner, which activates actors on demand. Single-writer is never violated because step 2 completes before step 3.

10. Remote transport

One ActorTransport interface, two implementations chosen by SLA. Messages serialize via FlexCoder (already exists) into a RemoteEnvelope.

interface ActorTransport {
  suspend fun<M : Any> tell(target: PodId, key: ActorKey<M>, message: M)
  suspend fun<M : Any, R> ask(target: PodId, key: ActorKey<M>, timeout: Duration, build: (Reply<R>) -> M): R
}

data class RemoteEnvelope(
  val messageId: String,         // UUID v7 — dedup key for idempotent receive
  val targetAddress: String,     // reaktor://region/scope/shard/key
  val orderingKey: String,       // == targetAddress  ⇒ per-actor FIFO
  val correlationId: String?,    // ask/reply routing
  val payload: ByteArray,        // FlexCoder-encoded M
  val senderReplyTo: String?,    // reply topic for ask
)

Hot vs durable selection: the cluster tries gRPC first for same-region targets (latency); on failure, timeout, or detected handoff it falls back to Pub/Sub (durability). A per-actor sequence number in the envelope lets the receiver detect/repair reordering when a sender switches transports mid-stream.

11. Virtual actors (Orleans grains)

"Across global instances" only scales if you don't pay for idle actors. Virtual actors make millions of identities cheap — only the active ones occupy memory.

Activation

First message to chats/room-42 with no live instance: the shard owner constructs the ActorNode, rehydrates state from ObjectStore (already exclusive + persisted), runs preStart(), delivers the message. Cold activation target: < 50 ms for small state.

Passivation

Idle past timeout (default 5 min): snapshot state to ObjectStore, run postStop(), cancel the coroutine scope, free memory. Maps cleanly onto the existing Lifecycle.Saving → Destroying transitions.

Migration

On rebalance or graceful drain: passivate on the old owner, activate on the new. The directory updates; in-flight messages buffered in Pub/Sub flow to the new owner. No state lost because state lives in ObjectStore, not memory.

Affinity

Placement hints: "activate in the user's home region" (data locality), "activate where the Memgraph replica is" (read locality), "co-locate with related actors" (chattiness). The coordinator honours hints when a shard has freedom.

Key reuse: the existing Graph.provides(key) { create } + findActorNode(key) already implement "get-or-create by stable key" — exactly the virtual-actor get-or-activate semantics, just locally. The cluster layer lifts this to "get-or-activate on the shard owner."

12. State & recovery

An actor can only move pods if its state is not trapped in one pod's memory. The existing exclusive ObjectStore already externalises state — but today it's backed by a node-local DB. For distribution it must be reachable from any owner.

• Externalise the store backend: actor ObjectStore for core actors writes to a shared backend — regional Postgres/Supabase (already wired) or a regional KV. Keyed by actors/<name> exactly as today. Any pod that becomes a shard's owner can open the same store.
• Snapshot model (default): passivation writes a full state snapshot; activation reads the latest. Simple, good for small/medium state.
• Event-sourcing (opt-in): high-write actors (chat rooms) append events to a per-actor journal + periodic snapshot; activation = latest snapshot + replay tail. Deterministic recovery and an audit log. Ties into the shadow env's capture model.
• Edge state: CF Durable Objects keep state in DO storage (already the case via room.storage). No externalisation needed — CF guarantees the DO is single-instance and its storage travels with it.

13. Distributed supervision

Local supervision already exists (ActorDirective). Distribution adds two things:

13.1 Node-level supervision

A pod is itself a supervised entity. Pod crash → failure detector fires → coordinator treats every shard the pod owned as "crashed children" and restarts them (rehydrate elsewhere). Restart-intensity limits apply at pod granularity: if a pod keeps crash-looping on activation of a poison actor, that actor is quarantined (dead-letter + alert) rather than taking down successive pods.

13.2 watch / monitor across the network

// An actor can monitor a remote actor and be told when it (or its node) dies.
interface ActorRef<M : Any> {
  // ... existing send / trySend / ask ...
  fun watch(watcher: ActorRef<*>)        // deliver Terminated(address, reason) to watcher
  fun unwatch(watcher: ActorRef<*>)
}

Implemented over the failure detector: when the coordinator marks a pod down, it synthesises Terminated signals for every watched actor on that pod and routes them to watchers. This is Erlang's monitor/2 — the foundation for "if the room authority dies, the presence actor cleans up."

13.3 Poison-message handling

A message that reliably crashes its actor would, with naive at-least-once + restart, loop forever. Mitigation: track per-message delivery attempts in the envelope; after N (default 5) the message is dead-lettered (captured, alerted) and the actor resumes. Mirrors the bus DLQ in the shadow + notifications designs.

14. Delivery & ordering

15. Split-brain & single-writer safety

The actor model's core invariant is one writer per actor. A network partition that makes two pods each believe they own a shard would violate it. Defences:

• Lease-based ownership: a pod may only host a shard while it holds the shard's lease (k8s Lease object, or a CF DO acting as the lock). Leases expire; a partitioned pod that can't renew loses ownership and stops processing — it cannot keep writing.
• Coordinator singleton via lease: only the lease-holder is the coordinator. A partition can't produce two coordinators handing out conflicting maps.
• Fencing tokens: each ownership grant carries a monotonic token; the ObjectStore backend rejects writes with a stale token. Even if a zombie pod thinks it owns a shard, its writes are fenced out.
• Edge is immune: Cloudflare guarantees a Durable Object is globally single-instance — edge actors get single-writer from the platform, no lease needed.

16. Cloudflare edge actors

Cloudflare Durable Objects are the closest thing in production to an Orleans grain: idFromName(name) maps any name to a globally-unique, single-instance, persistent object, routed to the nearest healthy colo. The PartyKitServer in reaktor-cloudflare already runs chat rooms this way.

16.1 DO as ActorRef

// jsMain — a DO-backed ActorRef. Same interface, CF supplies the runtime.
class EdgeActorRef<M : Any>(
  override val key: ActorKey<M>,
  private val namespace: DurableObjectNamespace,
) : ActorRef<M> {
  private val stub get() = namespace.get(namespace.idFromName(key.name))

  override suspend fun send(message: M) {
    stub.fetch("https://do/tell", RequestInit(method = "POST", body = encode(message)))
  }
  override suspend fun<R> ask(timeout: Duration, build: (Reply<R>) -> M): R =
    decode(stub.fetch("https://do/ask", …).body)
}

16.2 Edge ↔ core bridge

An edge chat-room DO (fan-out, presence) talks to its core authority actor (persistence, moderation, knowledge-graph) over a Pub/Sub topic or a CF service binding to the k3s ingress. The split:

• Edge DO: holds the WebSocket connections, broadcasts, presence — the things that must be one network hop from the user.
• Core ActorNode: holds durable history, runs moderation, updates the social graph in Memgraph — the things that are heavy and centralised.
• Alarms: DO alarm() maps onto the actor schedule/every timers; the codex actor doc already calls this out as the backend-adapter path.

17. Global topology

USERS (global)
│
┌─────────────┴─────────────┐
▼                           ▼
Cloudflare edge (colo)      Cloudflare edge (colo)
┌────────────────────┐      ┌────────────────────┐
│ Durable Objects    │      │ Durable Objects    │
│ chat rooms         │      │ chat rooms         │
│ presence/session   │      │ presence/session   │
└─────────┬──────────┘      └─────────┬──────────┘
          │ service binding / Pub/Sub │
┌─────────┴────────────────────────────┴─────────┐
▼                                                ▼
REGION: asia-south1                              REGION: us-central1
┌──────────────────────────┐                     ┌──────────────────────────┐
│ k3s StatefulSet          │                     │ k3s StatefulSet          │
│ reaktor-actor-0/1/2      │◀────Pub/Sub───────▶│ reaktor-actor-0/1/2      │
│ ShardCoordinator         │ (ordering key =     │ ShardCoordinator         │
│ core ActorNodes          │  address)           │ core ActorNodes          │
│ ┌────────┐ ┌────────┐    │                     │ ┌────────┐ ┌────────┐    │
│ │Memgraph│ │Postgres│    │                     │ │Memgraph│ │Postgres│    │
│ └────────┘ └────────┘    │                     │ └────────┘ └────────┘    │
└──────────────────────────┘                     └──────────────────────────┘
          │                                                │
          └────── global directory (CF DO) ────────────────┘
                 "which region owns user/entity X"

gRPC    ── hot path within a region (sub-ms)
Pub/Sub ── durable path across pods, across regions (ordered, at-least-once)
CF DO   ── edge runtime + global directory

18. k3s manifests

The actor cluster is a StatefulSet + headless Service — modelled on the existing cloud/k3s/manifests/memgraph.yaml:

# cloud/k3s/manifests/reaktor-actor.yaml  (sketch)
apiVersion: v1
kind: Service
metadata: { name: reaktor-actor }
spec:
  clusterIP: None            # headless ⇒ stable per-pod DNS
  selector: { app: reaktor-actor }
  ports: [{ name: grpc, port: 7900 }, { name: gossip, port: 7901 }]
---
apiVersion: apps/v1
kind: StatefulSet
metadata: { name: reaktor-actor }
spec:
  serviceName: reaktor-actor
  replicas: 3                  # up from today's single app-server Deployment
  podManagementPolicy: Parallel
  template:
    spec:
      terminationGracePeriodSeconds: 60   # time to passivate + hand off shards
      serviceAccountName: reaktor-actor   # RBAC: watch Endpoints + Lease
      containers:
        - name: reaktor-actor
          env:
            - { name: POD_ORDINAL,  valueFrom: { fieldRef: { fieldPath: metadata.name }}}
            - { name: REGION,       value: asia-south1 }
            - { name: SHARD_COUNT,  value: "256" }
          readinessProbe:                  # only route once shards are claimed
            httpGet: { path: /ready, port: 8080 }
          lifecycle:
            preStop: { exec: { command: ["/bin/actorctl", "drain"] }}   # graceful handoff

RBAC: the pod's ServiceAccount needs get/list/watch on endpoints and get/update on a Lease in its namespace. preStop drain is the graceful-leave hook so rolling deploys hand shards off before SIGKILL.

19. Multi-region

• Region = cluster = node group: each region is its own k3s StatefulSet with its own ShardCoordinator. Within a region, gRPC hot path. The "BEAM node group" boundary is the region.
• Cross-region: Pub/Sub topics bridge regions (it's a global service). An actor in asia-south1 sends to an actor in us-central1 by publishing with the target's address as ordering key; the target region's subscriber delivers.
• Home region & locality: each entity (user, room) has a home region recorded in the global directory (a CF DO — globally consistent, matches spec §13.2). Actors activate in their home region for data locality with the regional Memgraph/Postgres. The directory answers "where does user X live?" at the edge before any core hop.
• Follow-the-sun / failover: if a whole region goes down, the global directory reassigns its entities' home region; actors reactivate from cross-region-replicated ObjectStore. This is the node-down handoff at region granularity.

20. Observability

Everything emits into the same OTel pipeline as performance-observability §11/§17, with actor-specific signals:

• Metrics: actor.mailbox.depth (gauge), actor.message.duration (histogram), actor.activations / passivations (counter), actor.restarts (counter), shard.rebalance.duration, cluster.members (gauge), actor.dead_letters.
• Traces: a message's correlation id flows through the RemoteEnvelope so a single trace spans sender → transport → remote receive → downstream — exactly the causal chain the shadow capture records.
• Blueprint overlay: the supervision tree renders as a nested graph (per spec §13.3) — supervisors as containers, actors as children; a crashing actor flashes red and the restart strategy animates. The shard map renders as a pod×shard heat grid; a rebalance animates shards migrating between pods.
• Cluster console: live member list, per-pod shard count + load, failure-detector phi values, lease holders.

21. Chaos & correctness testing

Distribution bugs only appear under failure. Test them deliberately — and reuse the shadow dev environment so chaos runs against production-shaped data:

• Pod kill: kubectl delete pod reaktor-actor-1 mid-load; assert every actor it owned reactivates elsewhere, no message lost (Pub/Sub replay), no double-write (fencing).
• Partition: iptables-drop between two pods; assert the minority side loses leases and stops writing; assert single-writer holds.
• Rolling deploy: assert preStop drain hands off shards with zero dropped messages and sub-second actor unavailability.
• Poison message: inject a message that crashes its actor; assert it dead-letters after N attempts and the actor resumes (no crash loop taking down pods).
• Ordering under transport switch: force a gRPC→Pub/Sub fallback mid-stream; assert per-actor order is preserved via sequence numbers.
• Determinism: replay a captured chat-room message sequence (shadow capsule) against a freshly-activated actor; assert identical resulting state.

22. Phased plan

Phase 0~1 week

Wire format & externalised state

RemoteEnvelope + FlexCoder for actor messages. Make actor ObjectStore backend pluggable to a shared regional store. ClusterAddress. No behaviour change yet — local actors keep working. DoD: an actor message round-trips through FlexCoder encode/decode in a test.

Phase 1~2 weeks

Single-region membership + gRPC transport

StatefulSet + headless Service (3 replicas). k8s Endpoints watch for membership. phi-accrual failure detector over gRPC heartbeats. ActorTransport gRPC impl. DoD: 3 pods see each other; pod A can tell an actor pinned to pod B.

Phase 2~2 weeks

Sharding + coordinator + location-transparent ActorRef

ShardCoordinator (lease-protected singleton), HRW hashing, shard→pod map broadcast. ClusterActorRef.resolve. DoD: send reaches the right actor regardless of which pod the caller is on; killing a pod reassigns its shards.

Phase 3~1.5 weeks

Virtual actors

Activation on first message (rehydrate from ObjectStore), passivation on idle, migration on rebalance with message buffering. DoD: an idle actor frees memory and reactivates with full state on next message; survives a pod kill.

Phase 4~1.5 weeks

Pub/Sub durable path + idempotent receive

Pub/Sub ActorTransport with orderingKey = address; dedup by messageId; gRPC→Pub/Sub fallback + sequence repair. DoD: messages survive a mid-flight pod handoff with no loss and correct order.

Phase 5~1 week

Distributed supervision + watch/monitor

Node-down → synthesised Terminated; watch/unwatch; poison-message dead-lettering; restart-intensity at pod scale. DoD: a watcher is notified within the failure-detection window when a watched remote actor's pod dies.

Phase 6~2 weeks

Cloudflare edge actors under one ActorRef

EdgeActorRef over DO; edge↔core bridge for BestBuds chat (edge fan-out + core authority). DO alarms ↔ actor timers. DoD: a chat room runs as an edge DO talking to a core ChatRoomActor, both addressed via ActorRef.

Phase 7~1.5 weeks

Split-brain safety + multi-region

Lease-based ownership, fencing tokens, global directory on CF DO, cross-region Pub/Sub, home-region placement. DoD: a simulated partition never produces two writers; a region failover reactivates entities elsewhere.

Phase 8~1 week

Observability + chaos suite

Actor metrics + Blueprint supervision/shard overlays + cluster console. The chaos tests in §21 wired into CI against the shadow environment.

23. Risks & open questions

Risks

• Split-brain double-write. The cardinal sin of distributed actors. Mitigated by lease-based ownership + fencing tokens + CAP-consistency choice (§15). Must be chaos-tested before any stateful actor ships distributed.
• Pub/Sub latency on the hot path. Pub/Sub is ~tens-to-hundreds of ms — unacceptable for interactive sends. That's why gRPC is the same-region hot path and Pub/Sub is only the durable/cross-region fallback. Risk: fallback fires too eagerly and tanks latency. Mitigation: tune the gRPC failure threshold; keep hot actors co-located.
• Cold-activation latency. First message to a passivated actor pays a rehydrate. For chat this is fine; for latency-critical paths, keep-warm or predictive activation. Risk: thundering herd on mass reactivation after a region failover. Mitigation: rate-limit activations, prioritise by recency.
• Coordinator as bottleneck/SPOF. Singleton. Mitigated by lease failover (new coordinator elected in seconds) and by caching the shard map on every pod so steady-state routing doesn't touch the coordinator. Only rebalances need it.
• ObjectStore backend hot spots. Externalising state means a shared DB carries all actor reads/writes. Risk: it becomes the bottleneck Erlang's per-process heap never was. Mitigation: regional sharded stores, event-sourcing for write-heavy actors, edge actors keep state in DO storage.
• Rebalance storms. Flapping membership (a pod that's slow but not dead) could thrash shard assignments. Mitigation: phi-accrual hysteresis, minimum dwell time before reassign, rate-limited rebalancing.
• Two state models (JVM ObjectStore vs DO storage). Edge and core actors persist differently. Risk: subtle semantic drift. Mitigation: the ActorRef abstraction hides it, but the bridge contract (what's authoritative where) must be explicit per actor type.
• k3s single-node today. The whole design assumes a real multi-node cluster; bestbuds runs 1 replica now. Phase 1 requires actually provisioning a multi-node k3s (or GKE) — an infra prerequisite, not just code.

Open questions

• Coordinator lease backend: k8s Lease (in-cluster, simple) vs a CF DO (works across regions, but adds an edge dependency to a core-path operation)?
• Is gRPC worth it, or start Pub/Sub-only and add gRPC when latency demands? (Pub/Sub-only is simpler but adds 50–200 ms to every cross-pod send.)
• Which BestBuds actors are core vs edge? Chat room is clearly split; what about presence, typing, the knowledge-graph workers, ranking?
• Event-sourcing by default for all actors, or snapshot-by-default with opt-in journaling for write-heavy ones?
• Do we adopt an existing library (Akka/Pekko on JVM gives cluster-sharding off the shelf) or build native to keep the graph-kernel + KMP + edge unification? Akka would be JVM-only and wouldn't cover the CF DO edge runtime — likely build native, borrow algorithms.
• Home-region assignment policy: by user signup region, by first-activity geo, or by explicit data-residency requirement?
• How does this interact with the shadow capture — is a distributed actor message a capture-eligible envelope by default?

24. References

Real code this design builds on

• reaktor-graph/src/commonMain/.../core/node/ActorNode.kt — the local actor: mailbox, supervision, ObjectStore, ask/tell, stable keys.
• reaktor-google/src/commonMain/.../PubSub.kt + jvmMain/JvmPubSubAdapter.kt — Pub/Sub with orderingKey (the per-actor FIFO primitive).
• reaktor-cloudflare/.../DurableObjects.kt + PartyKit.kt — edge actor runtime (idFromName, blockConcurrencyWhile, storage, WS hibernation).
• reaktor-flexbuffer/.../FlexCoder.kt — wire serialization for remote envelopes.
• bestbuds/cloud/k3s/manifests/memgraph.yaml — the StatefulSet + headless Service template.
• bestbuds/modules/app/src/jsMain/.../cloudflare/chat/ChatService.kt — an edge actor that already exists (extends PartyKitServer).
• reaktor-graph/.../service/ServiceTransport.kt — the PUBSUB/QUEUE/PEER enums this design gives delivery to.

Spec & peer designs

• Reaktor Spec §13 — Distributed Actor Model (Orleans/Erlang vision over Mesh); this doc adapts it to the server substrate.
• Reaktor Spec §33.2 — Actor Model: Theory and Industrial Scale.
• reaktor-actor-model — the codex local-actor design that scoped distribution out; this doc is its distributed sequel.
• reaktor-notifications-design — outbox + idempotency-key, a shared dependency for effectively-once.
• shadow-dev-environment — chaos testing substrate + capture-eligible actor messages.
• performance-observability — the OTel pipeline actor metrics/traces flow into.
• graph-kernel-analysis — node / port / visitor foundation.

External — distributed actor systems

• Erlang/OTP Design Principles (supervision, gen_server)
• Microsoft Orleans — virtual actors / grains
• Akka Cluster Sharding
• GCP Pub/Sub message ordering (ordering keys)
• Cloudflare Durable Objects
• The φ Accrual Failure Detector (Hayashibara et al.)
• Rendezvous (highest-random-weight) hashing
• Kubernetes StatefulSet + headless Service

Doc lineage. Authored 2026-05-28. Grounded in the real ActorNode implementation + a fresh substrate survey (Pub/Sub ordering keys, FlexCoder, PartyKit/DO bindings, k3s manifests — all confirmed real; reaktor-bus, Mesh/WebRTC, and PUBSUB/QUEUE transports confirmed aspirational). Adapts Reaktor Spec §13's Orleans/Erlang model from its WebRTC-mesh substrate to the actual server substrate (GCP CE k3s + Pub/Sub + Cloudflare Durable Objects). Sequel to the local-only reaktor-actor-model design.

Related docs

• Actor Model - Use before adding actor behavior or moving chat/session workflows into actors.
• Reaktor Bus - Use before designing durable async events, jobs, fan-out, queues, Pub/Sub, outbox, DLQ, replay, or graph-visible delivery.
• Server - Use before adding any production backend primitive (resilience, messaging, data patterns) or designing the Spring/Workers server runtime.
• Multitenancy - Use before designing tenant isolation, adding a tenant-scoped store, wiring the tid claim, or building per-tenant quotas/residency/lifecycle; pairs with reaktor-server (high-scale tenancy) and auth-analysis (the membership model).
• Mega-Scale Strategy - Use for enterprise-adoption strategy, the open-source vs commercial split, and sequencing the distributed-tier roadmap without diluting the single-threaded-actor core.
• Graph Kernel - Use to keep the core language small while modules add vocabulary.