Tidemark — Reaktor Data Layer

Architecture reconciled: 5 October 2026, extending the 11 September design. Tidemark provides the data infrastructure for Reaktor and Reaktor applications: local state, databases, files, queries, caches, replication, durable delivery, live sessions, and the lifecycle of their data. It must compose with services, actors, FFI, UI, workflows, deployment, and observability through the same typed heterogeneous multiplex port graph used by the rest of Reaktor.

The design follows the concerns of Designing Data-Intensive Applications: reliability, scalability, maintainability, data models, storage, replication, partitioning, transactions, batch and stream processing, and the consequences of partial failure. Its organizing idea is logical data and its guarantees first; qualified implementations underneath.

Data layer reading map

All dedicated data chapters now live in the Data layer navigation section, in reading order. This page remains the architectural entry point and owns querying, consistency, sync and lifecycle; the focused chapters own implementation detail.

ReadChapterOwns
1This architecture and Data graphLogical model, authority, store families, queries and graph vocabulary
2Current storage APIs, MultitenancyPresent source, browser migration and scoped isolation
3Transactions and derived stateCommands, caches, ledgers, outbox, sagas and projection positions
4Bus, MeshDurable delivery and live channels
5Queues and Workflows, WorkCloudflare durable jobs and local scheduling boundaries
6Data inspectionProvider views, authority, bounded results and evidence
7Implementation and sources, BestBuds schema hardeningQualification, preprint coverage and dated application evidence

Shared Service contracts, auth, graph kernel and Operations retain their broader ownership. Application-specific authority and product behavior are described in Manna and CompanyOS; the three-application adoption matrix connects their Queue/Workflow use cases.

The preprint's topology/product/observation distinction and evidence requirements are reconciled in the Data graph. The source and retirement register identifies removed obsolete plans and retained private guidance.

1. System design

The data graph is the data vocabulary and projection of the main Reaktor graph. It connects logical entities, schemas, datasets, access patterns, operations, derived representations, and lifecycle obligations to actual stores and running components. Data contracts are graph-native definitions with typed descriptors and bindings. Their semantic meaning remains independent of UI/process activation; the same identities drive composition, analysis and tooling. This does not require a second graph-neutral kernel or a separate identity system.

                         Main Reaktor semantic graph
       UI / service / actor / workflow / FFI operation definitions
                         reads · writes · invokes
                 entities · datasets · access patterns
               derives · replicates · indexes · archives
                         qualified bindings
          ┌──────────────┬──────────────┬─────────────────┐
          │ device/local │ edge/region  │ specialized     │
          │ SQLite       │ Postgres    │ Memgraph        │
          │ files/cache  │ D1 / DO / KV │ ClickHouse      │
          │ offline log  │ R2           │ other adapters  │
          └──────────────┴──────────────┴─────────────────┘
           durable delivery: outbox · pubsub · queues · logs
           live delivery: WebSocket · WebRTC / SFU · TURN

These are selectable components, not a mandatory deployment. A local app can have one SQLite file. A small connected app can use Postgres for relational data, documents, vectors, search, and an outbox. Specialized infrastructure appears when an access pattern, guarantee, placement constraint, or measured bottleneck justifies it.

ComponentResponsibilityBoundary
Semantic kernel and data vocabularyStable identity, schemas, relationships, requirements, revisionsPure descriptions; declaring data opens no connections
Service contractsTyped operations, interaction protocols, errors, effectsSeparate from implementations and transports
Data access and planningTyped queries, mutations, dependency analysis, capability checking, pushdownProduces an explainable plan; executors perform effects
Store adaptersNative transactions, indexes, scans, change capture, physical schema evolutionPreserve or explicitly reject requested semantics
Consistency and coordination runtimesTransaction domains, ownership, conflict handling, fencing, reconciliationUse database/actor/replication mechanisms appropriate to the workload
Sync and materializationOffline state, replicas, indexes, views, cache updates, checkpointsMake direction, lag, authority, and rebuildability explicit
Bus, work, and workflow runtimesDurable events, work distribution, retries, timers, multi-step effectsOwn durable execution and delivery semantics
Mesh and live sessionsPresence, media, ephemeral state, live data transportPersistence happens through an explicit recording/commit operation
Lifecycle and operationsBackup, restore, archive, deletion, encryption, capacity, telemetryPlans with scoped evidence and execution receipts

What is unified, and what remains specialized

Reaktor unifies identity, contracts, composition, policy, planning, and evidence. It does not implement a replacement storage engine, distributed consensus algorithm, universal transaction manager, or media server as part of its semantic kernel. B-trees, LSM trees, MVCC, compaction, WAL, replication protocols, and physical indexes remain backend responsibilities. Their observable consequences—range-scan support, isolation, write amplification, recovery behavior, limits—enter adapter profiles and planning.

The kernel is small and immutable. Feature modules contribute an open vocabulary. A data-only consumer must not acquire UI, DI, actor scheduling, or a live graph runtime just to open a local store. See the unified system graph and graph substrate for the shared definition/revision and activation/epoch model.

2. One data graph, within the main graph

The graph describes both what data means and how the system uses it. A logical Document can have a SQL record, an R2 body, Memgraph relationships, an embedding, a search index entry, an offline projection, and an archival representation. Those objects are connected by explicit mappings and derivations. They are not assumed to be interchangeable copies.

Editor.save ─writes→ Document ─storedAs→ Postgres.document
                         ├──body→ R2.object
                         ├──links→ Memgraph.document_vertex
                         ├──derivedBy Embed(model@revision)→ VectorIndex
                         ├──indexedBy Search(analyzer@revision)→ SearchIndex
                         └──syncView(tenant, predicate)→ Device.SQLite
SearchResults ─reads→ Search(query@revision) ─reads→ SearchIndex
ArchiveWorkflow ─archives→ Document's declared dependency closure

Editor.save, Embed, and Search are ordinary operation definitions in the main graph, whether implemented in Kotlin, TypeScript, C++, SQL, or an external service. The data view uses their existing IDs. A data lineage edge can therefore connect a UI requirement to a service, an FFI model invocation, and the database fields that determine its output.

The catalogue is a portable projection of definitions, not a requirement to copy every application row into a central graph database. Runtime entity references, sampled lineage, partition instances, and execution evidence are scoped, paginated overlays. Large product graphs can live in Memgraph or another graph store and be queried through the same contracts. Offline clients carry only the definitions and policy scope they need.

Logical identity and physical identity

Use stable dataset, field, query, operation, and entity identities independent of engine, hostname, table name, or cloud account. An entity reference includes its logical dataset, tenant/security namespace, and key. A physical locator such as postgres://cluster/schema/table is a binding, not the entity's identity. Provider credentials remain outside graph snapshots.

Definitions have revisions; deployed bindings have revisions; activations have epochs; data has versions scoped to a consistency domain. Do not collapse these into one version counter. UUIDs identify objects; timestamps help describe time; neither establishes transaction order. A sequence is ordered only within its declared log/partition and epoch. Hybrid logical clocks can assist causal tracking, but do not independently establish cross-database atomicity or a consistent recovery point.

The Data Graph specification defines nodes, edges, access patterns, graph-to-runtime plans, and checks in more detail.

3. Datastore families and default bindings

Families organize contracts and deployment choices. Capabilities determine compatibility. A product may implement several families or facets, and two members of one family are not automatically substitutable.

“Default” below means the intended Reaktor choice for that role, not a claim that every extension or adapter is installed. Reaktor currently uses Supabase only as a Postgres host. Supabase Auth, Storage, Realtime, and its other products are not dependencies of this architecture.

Family or facetReaktor alignmentAlternate bindingsCompatibility boundary
Relational, including documentsPostgres; SQLite/SQLDelight locally; D1 for suitable edge SQL. DocumentDb<T> normally stores typed JSON in SQL, with indexed fields promoted as needed.MySQL, TiDB, CockroachDB, SQL Server; MongoDB through an explicitly selected document adapterSQL dialect, JSON operations, constraints, transaction isolation, migrations and CDC differ. A document API does not promise arbitrary Mongo operators.
Vector facet of relational dataVectorIndex<T> modeled on pgvector; Postgres is the default bindingPinecone, Qdrant, Milvus, Cloudflare Vectorize, or a vector facet of another storeDimension, model revision, distance metric, exact/ANN behavior, filtering, updates and read consistency are contractual. Provider-specific features are extra facets.
Search facet of relational dataSearchIndex<T> modeled on pg_search: fielded text queries, relevance, filtering and bounded resultsElasticsearch, OpenSearch, Typesense, MeilisearchAnalyzer/tokenizer, query semantics, ranking, pagination, highlighting and aggregation profiles need qualification. Rankings need not be identical.
Key-value and structured stateObjectStore, local cache, Cloudflare KV; DO storage through a stateful service bindingRedis/Valkey, DynamoDB, FoundationDB, RocksDB, LMDB, other embedded KV storesKey lookup, CAS, multi-key atomicity, TTL, durability, ordering and replication consistency vary. DO execution is an additional coordination capability.
GraphMemgraph through graph services/BoltNeo4j; other property-graph adapters when qualifiedTraversals, edge identity, query language subset, procedures, constraints and transactions. RDF/SPARQL is a separate optional profile.
Columnar / large-scale analytical and wide-columnClickHouse for analytical serving; Parquet/Iceberg on R2 plus a selected query engine for lake/archive workloadsBigQuery, Snowflake, Redshift, DuckDB, Trino; Cassandra, ScyllaDB, HBase, Bigtable for the wide-column access profileOne family with distinct analytical scan/aggregate and partition-key wide-column profiles. Those profiles do not imply each other.
Time-seriesA time-series profile on Postgres/TimescaleDB or ClickHouse when suitable; no mandatory extra serviceInfluxDB, VictoriaMetrics, Prometheus-compatible stores, other qualified enginesEvent time, out-of-order writes, labels/cardinality, retention, rollups, window semantics and late-data policy remain first-class.
Object and fileR2 remotely; platform files locally; immutable blobs and manifestsS3, GCS, Azure Blob, MinIO; filesystem adaptersConditional writes, multipart upload, checksums, range reads, versioning, listing, file/POSIX semantics and retention controls differ.
Durable event logTransactional outbox/change log; Pub/Sub for durable event distribution when selectedKafka, Redpanda, Pulsar, Kinesis; other replayable logsPub/Sub subscriptions and replay are not automatically a Kafka-style indefinitely retained partition log. Retention, ordering, cursors and replay scope must be declared.

Documents, vectors, and search are independent access interfaces backed by relational storage by default, so using all three does not require three services. Their independence also permits moving an expensive index without changing the authoritative entity model. pgvector supplies exact and approximate similarity search in Postgres; pg_search supplies search-oriented indexing and queries. Both are extensions: deployment must verify host/version support. Postgres native full-text search can be another explicitly selected search profile; it is not silently equivalent to pg_search/BM25.

The combined columnar family is a packaging decision. ClickHouse-style analytical columns and Cassandra-style wide-column partition models solve different access patterns; this distinction is also explained in ClickHouse's storage overview. An aggregate query cannot fall back to an unbounded Cassandra scan merely because both are in the same family.

Queues, ephemeral pubsub, actor coordination, workflows, WebRTC, SFUs, and TURN are data services, not extra database families. Their contracts sit alongside these stores and can be implemented by products that also expose storage.

Profiles instead of a lowest common denominator

An adapter advertises versioned profiles with explicit limits and scoped evidence: transaction scope/isolation, unique constraints, conditional writes, query operators, changefeeds, consistency, durability, retention, residency, availability and target support. Requirements belong to a dataset or operation, not just to a global database selection.

Binding resolution returns supported, explicitly emulated with a reviewed plan, or unsupported. Emulation includes its additional components, cost, failure modes and semantic limits. No silent downgrade from serializable to eventual, exact to approximate, durable to ephemeral, or transactional CAS to read-then-write.

Extensions remain typed: Elasticsearch aggregations or Pinecone-specific capabilities attach a provider requirement to the calling operation. They do not expand every app's base interface. Such a requirement marks the corresponding portability boundary in the graph.

Specialized temporal/bitemporal, RDF, geospatial, spatial raster, array/scientific, ledger/immutable-audit, vector-model serving, and POSIX filesystem semantics are not all baseline promises. Some can be extension profiles on the families above; others need new vocabulary and adapters. Unsupported is a legitimate result.

4. Service: one contract concept, several interaction protocols

The current Service abstraction is a strong ergonomic starting point: typed request/response DTOs, named handlers, interceptors, and HTTP adapters. Its existing suspend (Request) -> Response path is insufficient for the entire stack. Upgrade the contract, while leaving database engines, schedulers, transports, and native memory management in their own implementations.

A service is a named collection of typed operations. Each operation declares stable identity/revision, input/output/error schemas, interaction protocol, effects, and capability requirements. An implementation supplies behavior; a binding supplies location, transport, lifecycle and an execution policy. These are independent choices only where their contracts allow it.

interface Contract<C> {
    val id: Ref
    val revision: Revision
    val protocol: Ref
    val requirements: Set<Requirement>
}

interface Binder {
    fun <C> bind(contract: Contract<C>, choice: BindingChoice): BindingResult<C>
}

fun interface SyncCall<I, O, E> {
    fun call(input: I, context: CallContext): Outcome<O, E>
}

fun interface AsyncCall<I, O, E> {
    suspend fun call(input: I, context: CallContext): Outcome<O, E>
}

fun interface StreamCall<I, O, E> {
    suspend fun open(input: I, resume: Cursor?, context: CallContext):
        Outcome<Receiver<O, E>, E>
}

fun interface DuplexCall<Open, I, O, E> {
    suspend fun open(input: Open, context: CallContext):
        Outcome<DuplexSession<I, O, E>, E>
}

fun interface Publisher<T, E> {
    suspend fun publish(event: Event<T>, context: CallContext):
        Outcome<PublicationReceipt, E>
}

fun interface DurableCall<I, O, E> {
    suspend fun submit(input: I, context: CallContext): Outcome<Job<O, E>, E>
}

These are callable interfaces, not merely transport enum values. The complete specimen also defines receivers with bounded demand, senders and half-close, upload completion, subscriptions with ack/retry/dead-letter, job inspection/watch/cancellation, typed outcomes, context, and data interfaces. The Service design specifies their behavioral laws and migration from today's handlers.

Common app code keeps the same small shape: define a typed operation and a handler, then bind it locally or remotely. Generated HTTP helpers such as GetHandler can remain compatibility facades. The descriptor carries enough information to generate clients and graph ports. Most app authors never construct low-level sessions, profiles, or binding plans by hand.

Context, failures, and execution

Invocation context carries validated authority references, tenant scope, deadline budget, idempotency key, causation/tracing, and optional consistency requirements. Business DTOs do not become mutable bags of HTTP headers or native pointers. The receiver validates authority; possession of a graph ID is not authorization.

An outcome distinguishes a typed domain rejection from a runtime/transport failure. A failure also reports whether an effect was not started, applied, partially applied, or unknown. A timeout after committing a payment cannot safely mean “nothing happened.” Accepted publication, committed state, projected state, and device delivery are separate receipts/checkpoints.

Actors are an optional execution policy for serialized state ownership. Pure synchronous functions and native kernels can execute directly. A synchronous contract requires a binding that can actually return synchronously on that target; it cannot be transparently rebound to network RPC in a browser. suspend does not itself allocate a thread, guarantee nonblocking implementation, or provide durability.

FFI is a Service binding family

FFI shares Service identity, schemas, effects and interaction protocols. Its binding owns ABI version, codec, target availability, runtime generation, handle ownership, thread affinity, callback lifecycle and reentrancy. Keep those implementation details in reaktor-ffi; do not duplicate the public service concept.

Native sync calls can remain direct, without a mandatory coroutine, actor mailbox, serialization pass, or graph lookup. Borrowed memory is valid only for its declared scope. Async operations require owned or retained buffers until terminal completion, including cancellation acknowledgement. Runtime-scoped handles cannot be sent as raw pointers to remote peers. Process crashes and invalid native memory access are not universally recoverable structured errors. See FFI.

5. Data access, intra-store links, and inter-store queries

A semantic link carries a key mapping, cardinality, direction, integrity policy, authority, visibility rules and version expectations. Within one relational database, it may compile to a foreign key and join. Within a graph database, it may compile to a relationship and traversal. A JSON field, blob reference or search hit can point at the same logical entity without pretending that its storage representation is identical.

A cross-store reference is not a cross-store foreign key. Its contract states how dangling references, deletion, authorization and reconciliation work. An ownership edge may require restrict, cascade, detach, tombstone, or retained-reference behavior; the graph does not infer destructive cascades merely from reachability.

The query layer needs a typed expression/plan IR: selection, projection, stable ordering, pagination, joins, graph traversals, aggregates, text/vector predicates, and explicit native extensions. A plan includes estimated cardinality, pushdown boundaries, credentials/authority scope, consistency requirements, data movement limits, and the evidence behind estimates.

Execution strategyUseRequired visibility
Native pushdownJoin documents to owners in one Postgres transaction/snapshotNative semantics and index plan
Batched compositionRead candidate graph IDs, fetch authorized SQL rows in bounded batchesBatch limits, missing rows, versions, no hidden per-row RPC loop
Maintained materializationSearch, vector index, denormalized feed or analytical read modelSource checkpoint, transform revision, lag, rebuild path
Qualified federationAd hoc cross-store analysis where adapters support itTransfer/scan budget, snapshot limitations, spill and cancellation
Reject or request another planUnsupported operators or unacceptable cost/consistencyConcrete missing capability; no silent full scan

For “search documents linked to this project and show author names,” the planner can push text/vector filtering into Postgres, traverse Memgraph for membership, and batch author reads. At higher scale it can use a maintained membership projection. These plans have different freshness and cost; the chosen contract makes that visible. Cross-store pagination needs a stable plan/snapshot or declared best-effort cursor. Approximate vector recall and changed search ranking cannot be disguised as byte-identical results.

Caching tracks queries and their membership predicates, not only rows previously returned. Inserts, deletes, permission changes, relationship changes, schema/transform revisions and embedding-model changes can invalidate a result. Cache identity includes tenant/authority scope, query and parameters, schema/plan revision, and appropriate version tokens. TTL is a bounded-staleness policy, not a proof of consistency. Opaque native SQL or FFI code must declare dependencies conservatively or opt out of automatic optimization.

6. Transactions, authority, replication, and failure

A dataset specifies who may accept writes and what invariants must hold. Select a consistency profile per operation and coordination domain:

ProfileMechanismSuitable examples
Transactional authorityNative DB transaction with required isolation/constraintsInventory, ledgers, unique names, relational invariants
Serialized entity/room authorityDO or actor service plus qualified durable storage and ownership/fencingRoom state, game turn, entity state machine
Replicated consensus authorityBackend replication and a qualified client/read protocolMulti-node durable state with specified availability/consistency
Convergent replicated stateA named CRDT or application merge law, causal metadata and repairCollaborative sets/counters/doc structures that satisfy that law
Immutable append and derivationOrdered source log within declared scope plus idempotent transformsAudit streams, telemetry, analytical views

One app can use several profiles. A CRDT does not preserve arbitrary business invariants; an actor's local mailbox does not guarantee global single ownership; an HLC is not a consensus protocol. For cross-domain commands use a saga/workflow with explicit intermediate states and compensation, or an actually supported distributed transaction. Do not label a saga atomic.

Separate transaction domains, sync subscriptions, placement/sharding, tenant/security scopes, and retention/erasure groups. They often overlap but need not have the same boundary. For example, an offline dashboard may subscribe to several transactional accounts without making their updates one transaction.

Failure or pitfallRequired design
Commit succeeds but response is lostStable idempotency/effect ID, request fingerprint, durable result lookup; retry policy handles unknown outcomes
DB commit and event publish splitOutbox in the same transaction as state, or qualified CDC; relay and consumer deduplication
Duplicate or reordered deliveryIdempotent projector with checkpoint/version checks; ordering only within declared keys/partitions
Stale leader resumes after failoverFencing token checked by the resource that accepts writes; leases alone are insufficient
Network partitionPer-operation availability/consistency behavior; local pending state and explicit rejection/degradation
Retry storm or slow consumerDeadline budget, bounded queues/credits, jitter, retry ownership, DLQ and admission limits
Schema or code changes during replayVersioned events/mutations/transforms, compatible readers, pinned workflow history and migration plan
Unauthorized data survives in cacheScope-aware keys, revocation handling, local purge and repair, access checks at boundaries
Delete followed by old replay/restoreTombstone/deletion fence plus generation/checkpoint rules applied before serving restored data

The graph makes dependencies and obligations available to analysis. The mechanism and its conformance evidence supply the guarantee. Out-of-band writers, missing lineage and stale introspection reduce coverage and must appear as unknown or partial.

7. Sync, offline work, and reactive data

Sync is a family of protocols over declared datasets and views. A sync contract names source authority, selection predicate, direction, conflict law, schema revisions, checkpoint scheme, authorization, deletion behavior and resnapshot policy.

  • Read replication: initial snapshot plus ordered changes; appropriate for offline reference data and local read models.
  • Optimistic command/rebase: durable client outbox of named commands, speculative local state, authoritative execution, then rebase. Appropriate when replay is deterministic and side effects are separated.
  • CRDT or domain merge: selected for data with a proved/qualified merge law; tracks causal metadata and compaction conditions.
  • Bulk/batch synchronization: exports, CDC loads, ETL and periodic reconciliation with explicit freshness.
  • Live ephemeral state: presence/cursor/game prediction can use a live transport and periodic checkpoints without making every sample durable.

Snapshot and changefeed handoff must avoid gaps. Persist incoming changes and their cursor atomically where the profile requires it. Retain deduplication and tombstones long enough for supported offline/replay windows; an expired cursor requires resnapshot, not guessed continuation. Subscription changes and permission revocation include removals from the local view, not only newly visible rows.

A rebase transaction must never re-run email, payment, or network side effects while replaying local state. Those effects are named Service operations with durable execution/idempotency where required. Failed business commands stay inspectable; they do not disappear as successful local writes.

Device migration, logout, tenant switching, encryption key rotation and storage pressure are part of sync lifecycle. Browser persistence is a separate target qualification: an in-memory fallback is not durable SQLite/OPFS support. Reactive UI observes committed or explicitly speculative query results through native Flow/state bindings; collection cancellation releases its subscription.

8. Pubsub, queues, streams, and realtime

Bus is a first-class data-layer responsibility; a particular broker is optional. Publication describes facts/events. A queue distributes work among competing consumers. A retained log supports position-based replay. An ephemeral pubsub channel distributes live notifications. One product can support several, but the semantics must be selected explicitly.

RoleReaktor-aligned bindingContract
Domain event fanoutGCP Pub/Sub or another qualified brokerIndependent subscriptions, delivery/ack rules, ordering scope, retention and replay
Background workCloudflare Queues, platform work adapters, or another queueClaim/lease or delivery attempt, bounded retries, idempotent effect, DLQ and redrive
Source change streamTransactional outbox, DB CDC or retained logSource versions, snapshot handoff, replay horizon and schema
Live app channelWebSocket or WebRTC DataChannel through MeshConnection/session scope, ordering/reliability, bounded buffers, reconnect behavior
Live mediaCloudflare Realtime SFU or another SFUTrack/session lifecycle, congestion and participant authorization
Connectivity relayCloudflare TURN or another TURN serviceNAT traversal/relay; no application durability or event-log semantics

GCP Pub/Sub gives subscriptions independent delivery state. Cloudflare Queues uses at-least-once delivery. Applications still need their own effect identity and transaction boundary; broker guarantees do not automatically make database writes or external effects exactly once.

A subscription ack means the subscriber took responsibility according to its processing contract. It does not mean every user device received the event. Keep publication receipts, source commits, projection checkpoints and device cursors separate. CDC row changes and domain events can coexist; they are not necessarily the same schema or business meaning. Large payloads use authorized blob references with retention long enough for replay.

Mesh carries application data as well as audio/video. Cloudflare Realtime DataChannels can relay arbitrary application messages, while TURN provides connectivity relay. Reliable ordered delivery on an established connection is not durable storage across disconnects. Room/signaling services authorize participants and establish sessions; explicit commit/recording services persist durable facts.

A voice note normally becomes an R2 object plus a durable message record. A live call uses media tracks/SFU, optionally recording to object storage with a declared retention policy. Collaborative cursor movement can be ephemeral; an accepted document edit must follow the document's durable sync contract. See Bus and Mesh.

9. Durable workflows and materialized data

A durable Service call returns a job handle; the workflow runtime owns history, timers, retries, checkpoints and recovery. Workflow steps invoke typed Service operations and record durable effect identities. Cancellation is a request, not proof that external effects were undone. Versions pin code or supply an explicit compatible migration across suspended executions.

For an upload: reserve metadata, accept the object, verify checksum, invoke an FFI/native scanner or transcoder, generate thumbnails and embeddings, publish availability, and expire abandoned uploads. The graph relates each derived representation to its source, transform/model revision and checkpoint. It can identify stale thumbnails after a code change and schedule only affected rebuilds.

Materializations declare source authority, derivation, refresh mode, freshness budget, checkpoint, determinism assumptions and whether replay is sufficient to rebuild them. “Derived” does not mean disposable: external model output may be costly or nondeterministic, upstream logs may expire, and captured user events may be irreplaceable. Back up what cannot reliably be reconstructed.

Batch and streaming execution share logical transforms where semantics agree. Streaming additionally needs event-time versus processing-time rules, watermarks, late events, retractions/upserts, deduplication and bounded state. Backfills use separate epochs/checkpoints, run under resource budgets, and catch up before a versioned cutover. Kleppmann's data-infrastructure discussion motivates explicit logs and derived views rather than uncoordinated dual writes.

Cloudflare delivery and durable execution

Use Queues and Workflows as the canonical provider profile: durable intent and outbox relay, at-least-once/reordered commands, effect deduplication, Queue-to-Workflow handoff, checkpointed steps, waits, cancellation and lifecycle. These compose with the dataset's existing authority; neither a Queue nor a Workflow becomes the record store. CompanyOS has an existing Workflow skeleton, while shared Reaktor executors remain proposed. A run receipt and an applied source position establish different kinds of progress.

10. Backup, restore, archival, deletion, and encryption

Lifecycle plans traverse typed lifecycle relationships with policy, not every reachable graph edge. They account for authoritative records, blobs, replicas, derived indexes, caches, outboxes, broker retention, local devices and backups. A backup, an archive, and a replica have different purposes; none automatically substitutes for the others.

OperationHow the graph helpsWhat the runtime must do
BackupDiscover required representations, schemas, keys and dependency orderCapture native snapshots/log positions, checksums, manifests and encryption metadata; verify restorability
RestoreReconstruct dependencies and identify rebuildable indexesRestore into an isolated epoch, reconcile checkpoints, apply deletion fences, rebuild/catch up, validate before routing traffic
ArchiveSelect closed/cold data and known consumersProduce versioned immutable exports, preserve schema/provenance, verify reads before deleting hot copies
DeleteFind declared copies, references and derived contentExecute policy-specific purge/redaction/detach, collect receipts, report retained/unknown/unreachable copies
Key rotationLocate datasets and derivatives in each encryption domainRewrap or re-encrypt, migrate readers, track key versions and prove completion before retiring keys

A cross-store SnapshotSet manifest records each store's snapshot/log position, relevant schema/transform/binding revisions, blob versions/checksums, dependency closure and key references. A shared timestamp or HLC alone is insufficient. To claim application-consistent recovery, use coordinated barriers/quiescence or a log/checkpoint scheme that produces a causally valid cut. Otherwise label the backup crash-consistent or individually consistent and document reconciliation and potential loss. Declare RPO/RTO and test recovery against them.

Archival can use Parquet/Iceberg on R2, or another object/lake binding. An analytical serving engine such as ClickHouse is selected for query latency, concurrency and cost. No single engine, zero-cost floor, or dataset-size threshold is universal. Restoring old data must not resurrect deleted accounts; a protected deletion ledger/fence must outlive relevant backups and replay windows.

Encryption has distinct profiles: transport encryption, server-side encryption at rest, application field/blob encryption, and end-to-end encryption. The graph carries key references and protection domains, never plaintext secrets. It can reject an untrusted server transform over E2EE plaintext or route a permitted transform to a trusted client/runtime. Search and embeddings of encrypted data need an explicit trust and leakage model; ordinary server-side indexes do not magically work over opaque ciphertext.

Deleting a key only achieves the intended cryptographic erasure if all relevant copies, key backups, caches, exports and derivatives fall within that verified key boundary. Offline devices and unmanaged exports may be unreachable. Reports distinguish completed, pending, retained-by-policy, unsupported and unknown; traversal coverage is not a universal erasure or compliance proof.

11. Portability and changing infrastructure

The graph protects application meaning and dependency structure from changes in physical placement. It cannot erase differences in database behavior. Portability has several levels:

ChangeExpected disruptionWhat must be qualified
Supabase-hosted Postgres → RDS, another managed Postgres, or self-hosted PostgresUsually primarily binding and operations workExtension/version availability, privileges, connection pooling, networking, backups, CDC, failover and performance; no Supabase product migration is implied
Postgres → TiDBLogical entities and portable operations can survive; SQL/schema and affected features may changeTiDB is MySQL-compatible, not Postgres-compatible. Rewrite native SQL, JSON/DDL/type mappings and replace unavailable extension-backed facets.
Memgraph → Neo4jShared property-graph contracts can surviveCypher subset, procedures, indexes/constraints, identifiers, transactions and measured query behavior
pgvector/pg_search → dedicated vector/search enginesAuthoritative entities and callers of the qualified facet remain stableNew derivations/CDC, freshness, ranking/recall, filters, pagination, tenant isolation, index rebuild and rollout
Durable Object → Redis/Valkey plus computePotentially a runtime redesign behind the same stateful ServiceStorage alone does not replace actor ownership, placement, alarms, WebSockets or failure fencing. Compose and qualify the missing runtime components.
ClickHouse → ElasticsearchSome bounded event filtering/aggregation workloads may mapQuery semantics, joins, precision, scan/concurrency limits, freshness and cost; general OLAP replacement is not guaranteed
Cloudflare/GCP services → AWS equivalentsServices and data contracts can remain stable where profiles matchMap storage, compute, coordination, pubsub/queues, workflows, realtime, identity/secrets, observability and egress separately

TiDB's compatibility documentation and Memgraph's Cypher comparison make these distinctions concrete. Sharing SQL or Bolt is useful, but it is not a complete substitution proof.

A migration plan inventories consumers and native dependencies, compares required profiles, translates schema/query plans, backfills, captures ongoing changes, validates canonical data and query behavior, catches up to a checkpoint, and cuts over a binding revision. Avoid uncoordinated application dual writes. Rollback needs a data-compatibility and reverse-change plan once the new backend accepts writes; changing a connection string back is not sufficient.

For a Cloudflare-to-AWS migration, an R2 binding might become S3, a queue binding SQS, a fanout binding SNS/EventBridge with appropriate queue/log composition, and a service implementation Lambda or a container. A DO room authority may need compute plus a transactional store/ownership mechanism, and Mesh may need a different SFU/TURN provider. Those are candidate compositions, not certified one-to-one replacements. Placement and residency checks include logs, backups, caches, replicas and telemetry, not just the primary database.

12. Application scenarios

The following are design scenarios, not reports of implemented features.

ApplicationCompositionWhat the data layer contributes
Personal notes or local creative toolSQLite documents + files + reactive queries; optional syncSame logical document model works offline without brokers, remote graph services or distributed scheduling
Multi-tenant SaaS / CRMPostgres relational/JSON, pg_search, object attachments, transactional outboxShared tenancy/authorization across queries, files, caches and events; indexes can move independently later
Team knowledge and AI workspaceSQL documents, Memgraph links, vector/search facets, FFI model service, R2 sourcesTrace an answer/index hit to source and model revision; permission-aware retrieval; invalidate and rebuild affected derivatives
Chat, multiplayer or collaborative editorDurable messages/state, explicit merge/authority profile, local sync, Mesh presence/mediaDifferent durability for cursor samples, game state, messages and recordings; reconnect/replay without treating SFU delivery as persistence
Commerce or bookingTransactional inventory/accounting, order workflow, event projectionsProtect invariants inside the actual transaction domain; idempotent payment effects and visible compensation/reconciliation
Offline field or industrial appDevice database, scoped replication, sensor time-series, object evidencePending-versus-accepted state, bounded offline window, key/tenant lifecycle, reconnect repair, explicit conflict handling
Analytics and telemetry platformIngest log/queue, stream/batch transforms, ClickHouse, object archiveLate-event policy, backfill epochs, lineage, retention, resource budgets and reproducible rollups
Media/AI processing pipelineObject upload, durable jobs, native/FFI execution, generated assets and indexesShared service contracts connect compute to data; resumable effects, bounded progress streams and lifecycle dependency planning

For example, a CRM starts with customers and JSON notes in Postgres. Search and vectors stay in that database. Later, a recommendation feature uses Memgraph and a vector index. The existing Customer identity remains stable; new projections declare their source and lag. Deleting a customer can now plan removal from notes, attachments, recommendations, caches and indexes. That improves completeness only to the extent those dependencies are declared or observed.

A second example is a booking timeout: the database may have committed even though the client received no reply. The operation's idempotency key locates the prior result, and the outbox still delivers the booking event. A late projector uses source version checks; the UI can request at least the booking's commit token or display a pending projection. The graph connects these mechanisms; it does not replace any of them.

13. Overhead, cost, and scaling

There is no fixed $5/month architecture and no measured performance advantage claimed here. A static contract and a direct local binding can be very small. Distributed coordination, extra indexes, cross-store queries and retained history have real cost regardless of the abstraction used.

Source of overheadControl
Metadata/code generationShip scoped definitions; intern IDs; compile/cache plans by revision; load optional modules only when used
InvocationResolve bindings outside hot loops; direct sync calls where qualified; avoid forced actor/coroutine/serialization hops
Query planning/federationNative pushdown, prepared plans, bounded batches, explainable transfer/scan budgets; materialize only when justified
Write amplificationEvery outbox, replica, index and archive is an explicit write/storage cost; avoid indexing every representation by default
Consistency/coordinationPay for synchronous coordination only inside required domains; expose latency/availability consequences
Sync and recoveryBound logs, pending work and replay windows; compact only after safety conditions; budget rebuild/backfill capacity
OperationsAccount for adapter upgrades, conformance tests, schema evolution, debugging and incident ownership

Scaling evolves the bindings and plans while retaining logical identity:

  1. Local: one process, SQLite/files, direct Service and FFI calls.
  2. Small connected: Postgres with JSON/vector/search as needed, object storage, minimal work/outbox processing; use edge storage where its profile fits.
  3. Growing: add read models/replicas, broker-backed fanout, partitioned work, cache tiers and selected analytical/graph engines after measuring bottlenecks.
  4. Large or multi-region: partition by access and coordination requirements, isolate hot tenants/keys, choose regional write ownership or a qualified replicated backend, and govern cross-region movement and recovery.

The graph stores partitioning/routing intent; runtime directories, placement controllers and databases execute it. Repartitioning requires backfill, routing epochs and fencing, checkpoint catch-up, and a cutover protocol. A single hot key cannot be solved merely by adding partitions. Broad transactions and cross-shard joins remain coordination and data-movement costs. Avoid a global graph lookup or catalogue service on every request; distributed applications use revisioned local projections and controlled refresh.

Measure p50/p95/p99 latency, throughput, allocations, CPU, memory, connection use, scanned/transferred bytes, index growth, write amplification, replication/projector lag, queue age, replay duration, RPO/RTO and cost per workload/tenant. Compare identical workloads, consistency, data volume and concurrency against direct native adapters. Include cold start, slow consumers, skew, outages and recovery. No performance or savings claim becomes a fact before that evidence exists.

14. Compared with a conventional stack

DimensionReaktor designConventional application stack
CompositionCommon typed identities connect logic, data, deployment and lifecycleTeams often connect ORM, broker, cache, jobs and telemetry through separate conventions
PortabilityRequirements and native dependencies can be inspected before a swapDrivers/ORMs may abstract common operations; semantic dependencies often remain in app code
Distributed correctnessShared vocabulary enables reusable checks, failure tests and execution plansMature specialist tools offer strong mechanisms, but integration correctness remains application work
PerformanceCross-component context can enable pushdown, batching and targeted invalidationDirect integrations have fewer framework layers and mature engine-specific tuning
ComplexityMore work in framework contracts, adapters, planner and qualification; fewer repeated integration decisions for apps if done wellLess custom framework to maintain; more repeated glue and separate operational models
Adoption/debuggingRequires accurate declarations, transparent plans and strong escape hatchesFamiliar ecosystem, hiring, libraries and vendor tooling; fewer Reaktor-specific concepts

The strongest benefit is semantic continuity across an application's components. The strongest risk is creating a distributed platform whose abstractions promise more than its runtimes can enforce. Prefer small orthogonal contracts and explicit plans over a giant “universal database” interface. Keep native access available, recording the resulting requirements and unknown dependencies. Reaktor is worthwhile when shared composition and lifecycle knowledge repay the framework and adapter maintenance cost.

15. Implementation sequence and acceptance gates

The canonical delivery sequence and acceptance matrix own implementation gates. They cover semantic identity, Service protocols, store conformance, transactional intent, duplicates and recovery, sync, lifecycle, inspection and qualified provider changes. Follow one real vertical path before generalizing; the sequence is capability-driven rather than a fixed schedule.

A support claim names the exact adapter/provider version, target, operation/profile, configuration, test revision and scope. “Designed,” “source-observed,” “tested,” “measured,” “stale,” “partial,” and “unknown” remain distinct. A compiled interface is a useful starting artifact; it is not an implemented runtime.

Source grounding and companion documents

Current implementation boundaries were reviewed in reaktor-service (Service, RequestHandler, endpoints/interceptors), reaktor-ffi (Invokable, Switch, target bridges), and reaktor-db (object storage and JVM graph access). The current graph adapter's raw Cypher/driver types, the unary HTTP Service path and the partial FFI targets are migration work, not evidence of portability or protocol parity.

Continue with the Data Graph, Service, FFI, Bus, Mesh, Work, and unified system graph. This page is authoritative for data-layer scope, datastore families, and the distinction between shared contracts and qualified runtime guarantees.