Tidemark — Reaktor Data Layer
Architecture reconciled: 5 October 2026, extending the 11 September design. Tidemark provides the data infrastructure for Reaktor and Reaktor applications: local state, databases, files, queries, caches, replication, durable delivery, live sessions, and the lifecycle of their data. It must compose with services, actors, FFI, UI, workflows, deployment, and observability through the same typed heterogeneous multiplex port graph used by the rest of Reaktor.
The design follows the concerns of Designing Data-Intensive Applications: reliability, scalability, maintainability, data models, storage, replication, partitioning, transactions, batch and stream processing, and the consequences of partial failure. Its organizing idea is logical data and its guarantees first; qualified implementations underneath.
Data layer reading map
All dedicated data chapters now live in the Data layer navigation section, in reading order. This page remains the architectural entry point and owns querying, consistency, sync and lifecycle; the focused chapters own implementation detail.
| Read | Chapter | Owns |
|---|---|---|
| 1 | This architecture and Data graph | Logical model, authority, store families, queries and graph vocabulary |
| 2 | Current storage APIs, Multitenancy | Present source, browser migration and scoped isolation |
| 3 | Transactions and derived state | Commands, caches, ledgers, outbox, sagas and projection positions |
| 4 | Bus, Mesh | Durable delivery and live channels |
| 5 | Queues and Workflows, Work | Cloudflare durable jobs and local scheduling boundaries |
| 6 | Data inspection | Provider views, authority, bounded results and evidence |
| 7 | Implementation and sources, BestBuds schema hardening | Qualification, preprint coverage and dated application evidence |
Shared Service contracts, auth, graph kernel and Operations retain their broader ownership. Application-specific authority and product behavior are described in Manna and CompanyOS; the three-application adoption matrix connects their Queue/Workflow use cases.
The preprint's topology/product/observation distinction and evidence requirements are reconciled in the Data graph. The source and retirement register identifies removed obsolete plans and retained private guidance.
1. System design
The data graph is the data vocabulary and projection of the main Reaktor graph. It connects logical entities, schemas, datasets, access patterns, operations, derived representations, and lifecycle obligations to actual stores and running components. Data contracts are graph-native definitions with typed descriptors and bindings. Their semantic meaning remains independent of UI/process activation; the same identities drive composition, analysis and tooling. This does not require a second graph-neutral kernel or a separate identity system.
Main Reaktor semantic graph
UI / service / actor / workflow / FFI operation definitions
reads · writes · invokes
entities · datasets · access patterns
derives · replicates · indexes · archives
qualified bindings
┌──────────────┬──────────────┬─────────────────┐
│ device/local │ edge/region │ specialized │
│ SQLite │ Postgres │ Memgraph │
│ files/cache │ D1 / DO / KV │ ClickHouse │
│ offline log │ R2 │ other adapters │
└──────────────┴──────────────┴─────────────────┘
durable delivery: outbox · pubsub · queues · logs
live delivery: WebSocket · WebRTC / SFU · TURN
These are selectable components, not a mandatory deployment. A local app can have one SQLite file. A small connected app can use Postgres for relational data, documents, vectors, search, and an outbox. Specialized infrastructure appears when an access pattern, guarantee, placement constraint, or measured bottleneck justifies it.
| Component | Responsibility | Boundary |
|---|---|---|
| Semantic kernel and data vocabulary | Stable identity, schemas, relationships, requirements, revisions | Pure descriptions; declaring data opens no connections |
| Service contracts | Typed operations, interaction protocols, errors, effects | Separate from implementations and transports |
| Data access and planning | Typed queries, mutations, dependency analysis, capability checking, pushdown | Produces an explainable plan; executors perform effects |
| Store adapters | Native transactions, indexes, scans, change capture, physical schema evolution | Preserve or explicitly reject requested semantics |
| Consistency and coordination runtimes | Transaction domains, ownership, conflict handling, fencing, reconciliation | Use database/actor/replication mechanisms appropriate to the workload |
| Sync and materialization | Offline state, replicas, indexes, views, cache updates, checkpoints | Make direction, lag, authority, and rebuildability explicit |
| Bus, work, and workflow runtimes | Durable events, work distribution, retries, timers, multi-step effects | Own durable execution and delivery semantics |
| Mesh and live sessions | Presence, media, ephemeral state, live data transport | Persistence happens through an explicit recording/commit operation |
| Lifecycle and operations | Backup, restore, archive, deletion, encryption, capacity, telemetry | Plans with scoped evidence and execution receipts |
What is unified, and what remains specialized
Reaktor unifies identity, contracts, composition, policy, planning, and evidence. It does not implement a replacement storage engine, distributed consensus algorithm, universal transaction manager, or media server as part of its semantic kernel. B-trees, LSM trees, MVCC, compaction, WAL, replication protocols, and physical indexes remain backend responsibilities. Their observable consequences—range-scan support, isolation, write amplification, recovery behavior, limits—enter adapter profiles and planning.
The kernel is small and immutable. Feature modules contribute an open vocabulary. A data-only consumer must not acquire UI, DI, actor scheduling, or a live graph runtime just to open a local store. See the unified system graph and graph substrate for the shared definition/revision and activation/epoch model.
2. One data graph, within the main graph
The graph describes both what data means and how the system uses it. A logical Document can have a SQL record, an R2 body, Memgraph relationships, an embedding, a search index entry, an offline projection, and an archival representation. Those objects are connected by explicit mappings and derivations. They are not assumed to be interchangeable copies.
Editor.save ─writes→ Document ─storedAs→ Postgres.document
├──body→ R2.object
├──links→ Memgraph.document_vertex
├──derivedBy Embed(model@revision)→ VectorIndex
├──indexedBy Search(analyzer@revision)→ SearchIndex
└──syncView(tenant, predicate)→ Device.SQLite
SearchResults ─reads→ Search(query@revision) ─reads→ SearchIndex
ArchiveWorkflow ─archives→ Document's declared dependency closure
Editor.save, Embed, and Search are ordinary operation definitions in the main graph, whether implemented in Kotlin, TypeScript, C++, SQL, or an external service. The data view uses their existing IDs. A data lineage edge can therefore connect a UI requirement to a service, an FFI model invocation, and the database fields that determine its output.
The catalogue is a portable projection of definitions, not a requirement to copy every application row into a central graph database. Runtime entity references, sampled lineage, partition instances, and execution evidence are scoped, paginated overlays. Large product graphs can live in Memgraph or another graph store and be queried through the same contracts. Offline clients carry only the definitions and policy scope they need.
Logical identity and physical identity
Use stable dataset, field, query, operation, and entity identities independent of engine, hostname, table name, or cloud account. An entity reference includes its logical dataset, tenant/security namespace, and key. A physical locator such as postgres://cluster/schema/table is a binding, not the entity's identity. Provider credentials remain outside graph snapshots.
Definitions have revisions; deployed bindings have revisions; activations have epochs; data has versions scoped to a consistency domain. Do not collapse these into one version counter. UUIDs identify objects; timestamps help describe time; neither establishes transaction order. A sequence is ordered only within its declared log/partition and epoch. Hybrid logical clocks can assist causal tracking, but do not independently establish cross-database atomicity or a consistent recovery point.
The Data Graph specification defines nodes, edges, access patterns, graph-to-runtime plans, and checks in more detail.
3. Datastore families and default bindings
Families organize contracts and deployment choices. Capabilities determine compatibility. A product may implement several families or facets, and two members of one family are not automatically substitutable.
“Default” below means the intended Reaktor choice for that role, not a claim that every extension or adapter is installed. Reaktor currently uses Supabase only as a Postgres host. Supabase Auth, Storage, Realtime, and its other products are not dependencies of this architecture.
| Family or facet | Reaktor alignment | Alternate bindings | Compatibility boundary |
|---|---|---|---|
| Relational, including documents | Postgres; SQLite/SQLDelight locally; D1 for suitable edge SQL. DocumentDb<T> normally stores typed JSON in SQL, with indexed fields promoted as needed. | MySQL, TiDB, CockroachDB, SQL Server; MongoDB through an explicitly selected document adapter | SQL dialect, JSON operations, constraints, transaction isolation, migrations and CDC differ. A document API does not promise arbitrary Mongo operators. |
| Vector facet of relational data | VectorIndex<T> modeled on pgvector; Postgres is the default binding | Pinecone, Qdrant, Milvus, Cloudflare Vectorize, or a vector facet of another store | Dimension, model revision, distance metric, exact/ANN behavior, filtering, updates and read consistency are contractual. Provider-specific features are extra facets. |
| Search facet of relational data | SearchIndex<T> modeled on pg_search: fielded text queries, relevance, filtering and bounded results | Elasticsearch, OpenSearch, Typesense, Meilisearch | Analyzer/tokenizer, query semantics, ranking, pagination, highlighting and aggregation profiles need qualification. Rankings need not be identical. |
| Key-value and structured state | ObjectStore, local cache, Cloudflare KV; DO storage through a stateful service binding | Redis/Valkey, DynamoDB, FoundationDB, RocksDB, LMDB, other embedded KV stores | Key lookup, CAS, multi-key atomicity, TTL, durability, ordering and replication consistency vary. DO execution is an additional coordination capability. |
| Graph | Memgraph through graph services/Bolt | Neo4j; other property-graph adapters when qualified | Traversals, edge identity, query language subset, procedures, constraints and transactions. RDF/SPARQL is a separate optional profile. |
| Columnar / large-scale analytical and wide-column | ClickHouse for analytical serving; Parquet/Iceberg on R2 plus a selected query engine for lake/archive workloads | BigQuery, Snowflake, Redshift, DuckDB, Trino; Cassandra, ScyllaDB, HBase, Bigtable for the wide-column access profile | One family with distinct analytical scan/aggregate and partition-key wide-column profiles. Those profiles do not imply each other. |
| Time-series | A time-series profile on Postgres/TimescaleDB or ClickHouse when suitable; no mandatory extra service | InfluxDB, VictoriaMetrics, Prometheus-compatible stores, other qualified engines | Event time, out-of-order writes, labels/cardinality, retention, rollups, window semantics and late-data policy remain first-class. |
| Object and file | R2 remotely; platform files locally; immutable blobs and manifests | S3, GCS, Azure Blob, MinIO; filesystem adapters | Conditional writes, multipart upload, checksums, range reads, versioning, listing, file/POSIX semantics and retention controls differ. |
| Durable event log | Transactional outbox/change log; Pub/Sub for durable event distribution when selected | Kafka, Redpanda, Pulsar, Kinesis; other replayable logs | Pub/Sub subscriptions and replay are not automatically a Kafka-style indefinitely retained partition log. Retention, ordering, cursors and replay scope must be declared. |
Documents, vectors, and search are independent access interfaces backed by relational storage by default, so using all three does not require three services. Their independence also permits moving an expensive index without changing the authoritative entity model. pgvector supplies exact and approximate similarity search in Postgres; pg_search supplies search-oriented indexing and queries. Both are extensions: deployment must verify host/version support. Postgres native full-text search can be another explicitly selected search profile; it is not silently equivalent to pg_search/BM25.
The combined columnar family is a packaging decision. ClickHouse-style analytical columns and Cassandra-style wide-column partition models solve different access patterns; this distinction is also explained in ClickHouse's storage overview. An aggregate query cannot fall back to an unbounded Cassandra scan merely because both are in the same family.
Queues, ephemeral pubsub, actor coordination, workflows, WebRTC, SFUs, and TURN are data services, not extra database families. Their contracts sit alongside these stores and can be implemented by products that also expose storage.
Profiles instead of a lowest common denominator
An adapter advertises versioned profiles with explicit limits and scoped evidence: transaction scope/isolation, unique constraints, conditional writes, query operators, changefeeds, consistency, durability, retention, residency, availability and target support. Requirements belong to a dataset or operation, not just to a global database selection.
Binding resolution returns supported, explicitly emulated with a reviewed plan, or unsupported. Emulation includes its additional components, cost, failure modes and semantic limits. No silent downgrade from serializable to eventual, exact to approximate, durable to ephemeral, or transactional CAS to read-then-write.
Extensions remain typed: Elasticsearch aggregations or Pinecone-specific capabilities attach a provider requirement to the calling operation. They do not expand every app's base interface. Such a requirement marks the corresponding portability boundary in the graph.
Specialized temporal/bitemporal, RDF, geospatial, spatial raster, array/scientific, ledger/immutable-audit, vector-model serving, and POSIX filesystem semantics are not all baseline promises. Some can be extension profiles on the families above; others need new vocabulary and adapters. Unsupported is a legitimate result.
4. Service: one contract concept, several interaction protocols
The current Service abstraction is a strong ergonomic starting point: typed request/response DTOs, named handlers, interceptors, and HTTP adapters. Its existing suspend (Request) -> Response path is insufficient for the entire stack. Upgrade the contract, while leaving database engines, schedulers, transports, and native memory management in their own implementations.
A service is a named collection of typed operations. Each operation declares stable identity/revision, input/output/error schemas, interaction protocol, effects, and capability requirements. An implementation supplies behavior; a binding supplies location, transport, lifecycle and an execution policy. These are independent choices only where their contracts allow it.
interface Contract<C> {
val id: Ref
val revision: Revision
val protocol: Ref
val requirements: Set<Requirement>
}
interface Binder {
fun <C> bind(contract: Contract<C>, choice: BindingChoice): BindingResult<C>
}
fun interface SyncCall<I, O, E> {
fun call(input: I, context: CallContext): Outcome<O, E>
}
fun interface AsyncCall<I, O, E> {
suspend fun call(input: I, context: CallContext): Outcome<O, E>
}
fun interface StreamCall<I, O, E> {
suspend fun open(input: I, resume: Cursor?, context: CallContext):
Outcome<Receiver<O, E>, E>
}
fun interface DuplexCall<Open, I, O, E> {
suspend fun open(input: Open, context: CallContext):
Outcome<DuplexSession<I, O, E>, E>
}
fun interface Publisher<T, E> {
suspend fun publish(event: Event<T>, context: CallContext):
Outcome<PublicationReceipt, E>
}
fun interface DurableCall<I, O, E> {
suspend fun submit(input: I, context: CallContext): Outcome<Job<O, E>, E>
}
These are callable interfaces, not merely transport enum values. The complete specimen also defines receivers with bounded demand, senders and half-close, upload completion, subscriptions with ack/retry/dead-letter, job inspection/watch/cancellation, typed outcomes, context, and data interfaces. The Service design specifies their behavioral laws and migration from today's handlers.
Common app code keeps the same small shape: define a typed operation and a handler, then bind it locally or remotely. Generated HTTP helpers such as GetHandler can remain compatibility facades. The descriptor carries enough information to generate clients and graph ports. Most app authors never construct low-level sessions, profiles, or binding plans by hand.
Context, failures, and execution
Invocation context carries validated authority references, tenant scope, deadline budget, idempotency key, causation/tracing, and optional consistency requirements. Business DTOs do not become mutable bags of HTTP headers or native pointers. The receiver validates authority; possession of a graph ID is not authorization.
An outcome distinguishes a typed domain rejection from a runtime/transport failure. A failure also reports whether an effect was not started, applied, partially applied, or unknown. A timeout after committing a payment cannot safely mean “nothing happened.” Accepted publication, committed state, projected state, and device delivery are separate receipts/checkpoints.
Actors are an optional execution policy for serialized state ownership. Pure synchronous functions and native kernels can execute directly. A synchronous contract requires a binding that can actually return synchronously on that target; it cannot be transparently rebound to network RPC in a browser. suspend does not itself allocate a thread, guarantee nonblocking implementation, or provide durability.
FFI is a Service binding family
FFI shares Service identity, schemas, effects and interaction protocols. Its binding owns ABI version, codec, target availability, runtime generation, handle ownership, thread affinity, callback lifecycle and reentrancy. Keep those implementation details in reaktor-ffi; do not duplicate the public service concept.
Native sync calls can remain direct, without a mandatory coroutine, actor mailbox, serialization pass, or graph lookup. Borrowed memory is valid only for its declared scope. Async operations require owned or retained buffers until terminal completion, including cancellation acknowledgement. Runtime-scoped handles cannot be sent as raw pointers to remote peers. Process crashes and invalid native memory access are not universally recoverable structured errors. See FFI.
5. Data access, intra-store links, and inter-store queries
A semantic link carries a key mapping, cardinality, direction, integrity policy, authority, visibility rules and version expectations. Within one relational database, it may compile to a foreign key and join. Within a graph database, it may compile to a relationship and traversal. A JSON field, blob reference or search hit can point at the same logical entity without pretending that its storage representation is identical.
A cross-store reference is not a cross-store foreign key. Its contract states how dangling references, deletion, authorization and reconciliation work. An ownership edge may require restrict, cascade, detach, tombstone, or retained-reference behavior; the graph does not infer destructive cascades merely from reachability.
The query layer needs a typed expression/plan IR: selection, projection, stable ordering, pagination, joins, graph traversals, aggregates, text/vector predicates, and explicit native extensions. A plan includes estimated cardinality, pushdown boundaries, credentials/authority scope, consistency requirements, data movement limits, and the evidence behind estimates.
| Execution strategy | Use | Required visibility |
|---|---|---|
| Native pushdown | Join documents to owners in one Postgres transaction/snapshot | Native semantics and index plan |
| Batched composition | Read candidate graph IDs, fetch authorized SQL rows in bounded batches | Batch limits, missing rows, versions, no hidden per-row RPC loop |
| Maintained materialization | Search, vector index, denormalized feed or analytical read model | Source checkpoint, transform revision, lag, rebuild path |
| Qualified federation | Ad hoc cross-store analysis where adapters support it | Transfer/scan budget, snapshot limitations, spill and cancellation |
| Reject or request another plan | Unsupported operators or unacceptable cost/consistency | Concrete missing capability; no silent full scan |
For “search documents linked to this project and show author names,” the planner can push text/vector filtering into Postgres, traverse Memgraph for membership, and batch author reads. At higher scale it can use a maintained membership projection. These plans have different freshness and cost; the chosen contract makes that visible. Cross-store pagination needs a stable plan/snapshot or declared best-effort cursor. Approximate vector recall and changed search ranking cannot be disguised as byte-identical results.
Caching tracks queries and their membership predicates, not only rows previously returned. Inserts, deletes, permission changes, relationship changes, schema/transform revisions and embedding-model changes can invalidate a result. Cache identity includes tenant/authority scope, query and parameters, schema/plan revision, and appropriate version tokens. TTL is a bounded-staleness policy, not a proof of consistency. Opaque native SQL or FFI code must declare dependencies conservatively or opt out of automatic optimization.
6. Transactions, authority, replication, and failure
A dataset specifies who may accept writes and what invariants must hold. Select a consistency profile per operation and coordination domain:
| Profile | Mechanism | Suitable examples |
|---|---|---|
| Transactional authority | Native DB transaction with required isolation/constraints | Inventory, ledgers, unique names, relational invariants |
| Serialized entity/room authority | DO or actor service plus qualified durable storage and ownership/fencing | Room state, game turn, entity state machine |
| Replicated consensus authority | Backend replication and a qualified client/read protocol | Multi-node durable state with specified availability/consistency |
| Convergent replicated state | A named CRDT or application merge law, causal metadata and repair | Collaborative sets/counters/doc structures that satisfy that law |
| Immutable append and derivation | Ordered source log within declared scope plus idempotent transforms | Audit streams, telemetry, analytical views |
One app can use several profiles. A CRDT does not preserve arbitrary business invariants; an actor's local mailbox does not guarantee global single ownership; an HLC is not a consensus protocol. For cross-domain commands use a saga/workflow with explicit intermediate states and compensation, or an actually supported distributed transaction. Do not label a saga atomic.
Separate transaction domains, sync subscriptions, placement/sharding, tenant/security scopes, and retention/erasure groups. They often overlap but need not have the same boundary. For example, an offline dashboard may subscribe to several transactional accounts without making their updates one transaction.
| Failure or pitfall | Required design |
|---|---|
| Commit succeeds but response is lost | Stable idempotency/effect ID, request fingerprint, durable result lookup; retry policy handles unknown outcomes |
| DB commit and event publish split | Outbox in the same transaction as state, or qualified CDC; relay and consumer deduplication |
| Duplicate or reordered delivery | Idempotent projector with checkpoint/version checks; ordering only within declared keys/partitions |
| Stale leader resumes after failover | Fencing token checked by the resource that accepts writes; leases alone are insufficient |
| Network partition | Per-operation availability/consistency behavior; local pending state and explicit rejection/degradation |
| Retry storm or slow consumer | Deadline budget, bounded queues/credits, jitter, retry ownership, DLQ and admission limits |
| Schema or code changes during replay | Versioned events/mutations/transforms, compatible readers, pinned workflow history and migration plan |
| Unauthorized data survives in cache | Scope-aware keys, revocation handling, local purge and repair, access checks at boundaries |
| Delete followed by old replay/restore | Tombstone/deletion fence plus generation/checkpoint rules applied before serving restored data |
The graph makes dependencies and obligations available to analysis. The mechanism and its conformance evidence supply the guarantee. Out-of-band writers, missing lineage and stale introspection reduce coverage and must appear as unknown or partial.
7. Sync, offline work, and reactive data
Sync is a family of protocols over declared datasets and views. A sync contract names source authority, selection predicate, direction, conflict law, schema revisions, checkpoint scheme, authorization, deletion behavior and resnapshot policy.
- Read replication: initial snapshot plus ordered changes; appropriate for offline reference data and local read models.
- Optimistic command/rebase: durable client outbox of named commands, speculative local state, authoritative execution, then rebase. Appropriate when replay is deterministic and side effects are separated.
- CRDT or domain merge: selected for data with a proved/qualified merge law; tracks causal metadata and compaction conditions.
- Bulk/batch synchronization: exports, CDC loads, ETL and periodic reconciliation with explicit freshness.
- Live ephemeral state: presence/cursor/game prediction can use a live transport and periodic checkpoints without making every sample durable.
Snapshot and changefeed handoff must avoid gaps. Persist incoming changes and their cursor atomically where the profile requires it. Retain deduplication and tombstones long enough for supported offline/replay windows; an expired cursor requires resnapshot, not guessed continuation. Subscription changes and permission revocation include removals from the local view, not only newly visible rows.
A rebase transaction must never re-run email, payment, or network side effects while replaying local state. Those effects are named Service operations with durable execution/idempotency where required. Failed business commands stay inspectable; they do not disappear as successful local writes.
Device migration, logout, tenant switching, encryption key rotation and storage pressure are part of sync lifecycle. Browser persistence is a separate target qualification: an in-memory fallback is not durable SQLite/OPFS support. Reactive UI observes committed or explicitly speculative query results through native Flow/state bindings; collection cancellation releases its subscription.
8. Pubsub, queues, streams, and realtime
Bus is a first-class data-layer responsibility; a particular broker is optional. Publication describes facts/events. A queue distributes work among competing consumers. A retained log supports position-based replay. An ephemeral pubsub channel distributes live notifications. One product can support several, but the semantics must be selected explicitly.
| Role | Reaktor-aligned binding | Contract |
|---|---|---|
| Domain event fanout | GCP Pub/Sub or another qualified broker | Independent subscriptions, delivery/ack rules, ordering scope, retention and replay |
| Background work | Cloudflare Queues, platform work adapters, or another queue | Claim/lease or delivery attempt, bounded retries, idempotent effect, DLQ and redrive |
| Source change stream | Transactional outbox, DB CDC or retained log | Source versions, snapshot handoff, replay horizon and schema |
| Live app channel | WebSocket or WebRTC DataChannel through Mesh | Connection/session scope, ordering/reliability, bounded buffers, reconnect behavior |
| Live media | Cloudflare Realtime SFU or another SFU | Track/session lifecycle, congestion and participant authorization |
| Connectivity relay | Cloudflare TURN or another TURN service | NAT traversal/relay; no application durability or event-log semantics |
GCP Pub/Sub gives subscriptions independent delivery state. Cloudflare Queues uses at-least-once delivery. Applications still need their own effect identity and transaction boundary; broker guarantees do not automatically make database writes or external effects exactly once.
A subscription ack means the subscriber took responsibility according to its processing contract. It does not mean every user device received the event. Keep publication receipts, source commits, projection checkpoints and device cursors separate. CDC row changes and domain events can coexist; they are not necessarily the same schema or business meaning. Large payloads use authorized blob references with retention long enough for replay.
Mesh carries application data as well as audio/video. Cloudflare Realtime DataChannels can relay arbitrary application messages, while TURN provides connectivity relay. Reliable ordered delivery on an established connection is not durable storage across disconnects. Room/signaling services authorize participants and establish sessions; explicit commit/recording services persist durable facts.
A voice note normally becomes an R2 object plus a durable message record. A live call uses media tracks/SFU, optionally recording to object storage with a declared retention policy. Collaborative cursor movement can be ephemeral; an accepted document edit must follow the document's durable sync contract. See Bus and Mesh.
9. Durable workflows and materialized data
A durable Service call returns a job handle; the workflow runtime owns history, timers, retries, checkpoints and recovery. Workflow steps invoke typed Service operations and record durable effect identities. Cancellation is a request, not proof that external effects were undone. Versions pin code or supply an explicit compatible migration across suspended executions.
For an upload: reserve metadata, accept the object, verify checksum, invoke an FFI/native scanner or transcoder, generate thumbnails and embeddings, publish availability, and expire abandoned uploads. The graph relates each derived representation to its source, transform/model revision and checkpoint. It can identify stale thumbnails after a code change and schedule only affected rebuilds.
Materializations declare source authority, derivation, refresh mode, freshness budget, checkpoint, determinism assumptions and whether replay is sufficient to rebuild them. “Derived” does not mean disposable: external model output may be costly or nondeterministic, upstream logs may expire, and captured user events may be irreplaceable. Back up what cannot reliably be reconstructed.
Batch and streaming execution share logical transforms where semantics agree. Streaming additionally needs event-time versus processing-time rules, watermarks, late events, retractions/upserts, deduplication and bounded state. Backfills use separate epochs/checkpoints, run under resource budgets, and catch up before a versioned cutover. Kleppmann's data-infrastructure discussion motivates explicit logs and derived views rather than uncoordinated dual writes.
Cloudflare delivery and durable execution
Use Queues and Workflows as the canonical provider profile: durable intent and outbox relay, at-least-once/reordered commands, effect deduplication, Queue-to-Workflow handoff, checkpointed steps, waits, cancellation and lifecycle. These compose with the dataset's existing authority; neither a Queue nor a Workflow becomes the record store. CompanyOS has an existing Workflow skeleton, while shared Reaktor executors remain proposed. A run receipt and an applied source position establish different kinds of progress.
10. Backup, restore, archival, deletion, and encryption
Lifecycle plans traverse typed lifecycle relationships with policy, not every reachable graph edge. They account for authoritative records, blobs, replicas, derived indexes, caches, outboxes, broker retention, local devices and backups. A backup, an archive, and a replica have different purposes; none automatically substitutes for the others.
| Operation | How the graph helps | What the runtime must do |
|---|---|---|
| Backup | Discover required representations, schemas, keys and dependency order | Capture native snapshots/log positions, checksums, manifests and encryption metadata; verify restorability |
| Restore | Reconstruct dependencies and identify rebuildable indexes | Restore into an isolated epoch, reconcile checkpoints, apply deletion fences, rebuild/catch up, validate before routing traffic |
| Archive | Select closed/cold data and known consumers | Produce versioned immutable exports, preserve schema/provenance, verify reads before deleting hot copies |
| Delete | Find declared copies, references and derived content | Execute policy-specific purge/redaction/detach, collect receipts, report retained/unknown/unreachable copies |
| Key rotation | Locate datasets and derivatives in each encryption domain | Rewrap or re-encrypt, migrate readers, track key versions and prove completion before retiring keys |
A cross-store SnapshotSet manifest records each store's snapshot/log position, relevant schema/transform/binding revisions, blob versions/checksums, dependency closure and key references. A shared timestamp or HLC alone is insufficient. To claim application-consistent recovery, use coordinated barriers/quiescence or a log/checkpoint scheme that produces a causally valid cut. Otherwise label the backup crash-consistent or individually consistent and document reconciliation and potential loss. Declare RPO/RTO and test recovery against them.
Archival can use Parquet/Iceberg on R2, or another object/lake binding. An analytical serving engine such as ClickHouse is selected for query latency, concurrency and cost. No single engine, zero-cost floor, or dataset-size threshold is universal. Restoring old data must not resurrect deleted accounts; a protected deletion ledger/fence must outlive relevant backups and replay windows.
Encryption has distinct profiles: transport encryption, server-side encryption at rest, application field/blob encryption, and end-to-end encryption. The graph carries key references and protection domains, never plaintext secrets. It can reject an untrusted server transform over E2EE plaintext or route a permitted transform to a trusted client/runtime. Search and embeddings of encrypted data need an explicit trust and leakage model; ordinary server-side indexes do not magically work over opaque ciphertext.
Deleting a key only achieves the intended cryptographic erasure if all relevant copies, key backups, caches, exports and derivatives fall within that verified key boundary. Offline devices and unmanaged exports may be unreachable. Reports distinguish completed, pending, retained-by-policy, unsupported and unknown; traversal coverage is not a universal erasure or compliance proof.
11. Portability and changing infrastructure
The graph protects application meaning and dependency structure from changes in physical placement. It cannot erase differences in database behavior. Portability has several levels:
| Change | Expected disruption | What must be qualified |
|---|---|---|
| Supabase-hosted Postgres → RDS, another managed Postgres, or self-hosted Postgres | Usually primarily binding and operations work | Extension/version availability, privileges, connection pooling, networking, backups, CDC, failover and performance; no Supabase product migration is implied |
| Postgres → TiDB | Logical entities and portable operations can survive; SQL/schema and affected features may change | TiDB is MySQL-compatible, not Postgres-compatible. Rewrite native SQL, JSON/DDL/type mappings and replace unavailable extension-backed facets. |
| Memgraph → Neo4j | Shared property-graph contracts can survive | Cypher subset, procedures, indexes/constraints, identifiers, transactions and measured query behavior |
| pgvector/pg_search → dedicated vector/search engines | Authoritative entities and callers of the qualified facet remain stable | New derivations/CDC, freshness, ranking/recall, filters, pagination, tenant isolation, index rebuild and rollout |
| Durable Object → Redis/Valkey plus compute | Potentially a runtime redesign behind the same stateful Service | Storage alone does not replace actor ownership, placement, alarms, WebSockets or failure fencing. Compose and qualify the missing runtime components. |
| ClickHouse → Elasticsearch | Some bounded event filtering/aggregation workloads may map | Query semantics, joins, precision, scan/concurrency limits, freshness and cost; general OLAP replacement is not guaranteed |
| Cloudflare/GCP services → AWS equivalents | Services and data contracts can remain stable where profiles match | Map storage, compute, coordination, pubsub/queues, workflows, realtime, identity/secrets, observability and egress separately |
TiDB's compatibility documentation and Memgraph's Cypher comparison make these distinctions concrete. Sharing SQL or Bolt is useful, but it is not a complete substitution proof.
A migration plan inventories consumers and native dependencies, compares required profiles, translates schema/query plans, backfills, captures ongoing changes, validates canonical data and query behavior, catches up to a checkpoint, and cuts over a binding revision. Avoid uncoordinated application dual writes. Rollback needs a data-compatibility and reverse-change plan once the new backend accepts writes; changing a connection string back is not sufficient.
For a Cloudflare-to-AWS migration, an R2 binding might become S3, a queue binding SQS, a fanout binding SNS/EventBridge with appropriate queue/log composition, and a service implementation Lambda or a container. A DO room authority may need compute plus a transactional store/ownership mechanism, and Mesh may need a different SFU/TURN provider. Those are candidate compositions, not certified one-to-one replacements. Placement and residency checks include logs, backups, caches, replicas and telemetry, not just the primary database.
12. Application scenarios
The following are design scenarios, not reports of implemented features.
| Application | Composition | What the data layer contributes |
|---|---|---|
| Personal notes or local creative tool | SQLite documents + files + reactive queries; optional sync | Same logical document model works offline without brokers, remote graph services or distributed scheduling |
| Multi-tenant SaaS / CRM | Postgres relational/JSON, pg_search, object attachments, transactional outbox | Shared tenancy/authorization across queries, files, caches and events; indexes can move independently later |
| Team knowledge and AI workspace | SQL documents, Memgraph links, vector/search facets, FFI model service, R2 sources | Trace an answer/index hit to source and model revision; permission-aware retrieval; invalidate and rebuild affected derivatives |
| Chat, multiplayer or collaborative editor | Durable messages/state, explicit merge/authority profile, local sync, Mesh presence/media | Different durability for cursor samples, game state, messages and recordings; reconnect/replay without treating SFU delivery as persistence |
| Commerce or booking | Transactional inventory/accounting, order workflow, event projections | Protect invariants inside the actual transaction domain; idempotent payment effects and visible compensation/reconciliation |
| Offline field or industrial app | Device database, scoped replication, sensor time-series, object evidence | Pending-versus-accepted state, bounded offline window, key/tenant lifecycle, reconnect repair, explicit conflict handling |
| Analytics and telemetry platform | Ingest log/queue, stream/batch transforms, ClickHouse, object archive | Late-event policy, backfill epochs, lineage, retention, resource budgets and reproducible rollups |
| Media/AI processing pipeline | Object upload, durable jobs, native/FFI execution, generated assets and indexes | Shared service contracts connect compute to data; resumable effects, bounded progress streams and lifecycle dependency planning |
For example, a CRM starts with customers and JSON notes in Postgres. Search and vectors stay in that database. Later, a recommendation feature uses Memgraph and a vector index. The existing Customer identity remains stable; new projections declare their source and lag. Deleting a customer can now plan removal from notes, attachments, recommendations, caches and indexes. That improves completeness only to the extent those dependencies are declared or observed.
A second example is a booking timeout: the database may have committed even though the client received no reply. The operation's idempotency key locates the prior result, and the outbox still delivers the booking event. A late projector uses source version checks; the UI can request at least the booking's commit token or display a pending projection. The graph connects these mechanisms; it does not replace any of them.
13. Overhead, cost, and scaling
There is no fixed $5/month architecture and no measured performance advantage claimed here. A static contract and a direct local binding can be very small. Distributed coordination, extra indexes, cross-store queries and retained history have real cost regardless of the abstraction used.
| Source of overhead | Control |
|---|---|
| Metadata/code generation | Ship scoped definitions; intern IDs; compile/cache plans by revision; load optional modules only when used |
| Invocation | Resolve bindings outside hot loops; direct sync calls where qualified; avoid forced actor/coroutine/serialization hops |
| Query planning/federation | Native pushdown, prepared plans, bounded batches, explainable transfer/scan budgets; materialize only when justified |
| Write amplification | Every outbox, replica, index and archive is an explicit write/storage cost; avoid indexing every representation by default |
| Consistency/coordination | Pay for synchronous coordination only inside required domains; expose latency/availability consequences |
| Sync and recovery | Bound logs, pending work and replay windows; compact only after safety conditions; budget rebuild/backfill capacity |
| Operations | Account for adapter upgrades, conformance tests, schema evolution, debugging and incident ownership |
Scaling evolves the bindings and plans while retaining logical identity:
- Local: one process, SQLite/files, direct Service and FFI calls.
- Small connected: Postgres with JSON/vector/search as needed, object storage, minimal work/outbox processing; use edge storage where its profile fits.
- Growing: add read models/replicas, broker-backed fanout, partitioned work, cache tiers and selected analytical/graph engines after measuring bottlenecks.
- Large or multi-region: partition by access and coordination requirements, isolate hot tenants/keys, choose regional write ownership or a qualified replicated backend, and govern cross-region movement and recovery.
The graph stores partitioning/routing intent; runtime directories, placement controllers and databases execute it. Repartitioning requires backfill, routing epochs and fencing, checkpoint catch-up, and a cutover protocol. A single hot key cannot be solved merely by adding partitions. Broad transactions and cross-shard joins remain coordination and data-movement costs. Avoid a global graph lookup or catalogue service on every request; distributed applications use revisioned local projections and controlled refresh.
Measure p50/p95/p99 latency, throughput, allocations, CPU, memory, connection use, scanned/transferred bytes, index growth, write amplification, replication/projector lag, queue age, replay duration, RPO/RTO and cost per workload/tenant. Compare identical workloads, consistency, data volume and concurrency against direct native adapters. Include cold start, slow consumers, skew, outages and recovery. No performance or savings claim becomes a fact before that evidence exists.
14. Compared with a conventional stack
| Dimension | Reaktor design | Conventional application stack |
|---|---|---|
| Composition | Common typed identities connect logic, data, deployment and lifecycle | Teams often connect ORM, broker, cache, jobs and telemetry through separate conventions |
| Portability | Requirements and native dependencies can be inspected before a swap | Drivers/ORMs may abstract common operations; semantic dependencies often remain in app code |
| Distributed correctness | Shared vocabulary enables reusable checks, failure tests and execution plans | Mature specialist tools offer strong mechanisms, but integration correctness remains application work |
| Performance | Cross-component context can enable pushdown, batching and targeted invalidation | Direct integrations have fewer framework layers and mature engine-specific tuning |
| Complexity | More work in framework contracts, adapters, planner and qualification; fewer repeated integration decisions for apps if done well | Less custom framework to maintain; more repeated glue and separate operational models |
| Adoption/debugging | Requires accurate declarations, transparent plans and strong escape hatches | Familiar ecosystem, hiring, libraries and vendor tooling; fewer Reaktor-specific concepts |
The strongest benefit is semantic continuity across an application's components. The strongest risk is creating a distributed platform whose abstractions promise more than its runtimes can enforce. Prefer small orthogonal contracts and explicit plans over a giant “universal database” interface. Keep native access available, recording the resulting requirements and unknown dependencies. Reaktor is worthwhile when shared composition and lifecycle knowledge repay the framework and adapter maintenance cost.
15. Implementation sequence and acceptance gates
The canonical delivery sequence and acceptance matrix own implementation gates. They cover semantic identity, Service protocols, store conformance, transactional intent, duplicates and recovery, sync, lifecycle, inspection and qualified provider changes. Follow one real vertical path before generalizing; the sequence is capability-driven rather than a fixed schedule.
A support claim names the exact adapter/provider version, target, operation/profile, configuration, test revision and scope. “Designed,” “source-observed,” “tested,” “measured,” “stale,” “partial,” and “unknown” remain distinct. A compiled interface is a useful starting artifact; it is not an implemented runtime.
Source grounding and companion documents
Current implementation boundaries were reviewed in reaktor-service (Service, RequestHandler, endpoints/interceptors), reaktor-ffi (Invokable, Switch, target bridges), and reaktor-db (object storage and JVM graph access). The current graph adapter's raw Cypher/driver types, the unary HTTP Service path and the partial FFI targets are migration work, not evidence of portability or protocol parity.
Continue with the Data Graph, Service, FFI, Bus, Mesh, Work, and unified system graph. This page is authoritative for data-layer scope, datastore families, and the distinction between shared contracts and qualified runtime guarantees.