Migrating Korivo to a shared Rust core

The problem with having two correct implementations

Korivo is designed to be local-first. Records live on a person’s devices and, when synchronization is enabled, in a folder controlled through their storage provider. There is no Korivo service that serializes concurrent writes or provides a canonical database state. Each application has to interpret the records it can observe and independently reach the same result.

Consider a fairly ordinary sequence. Someone records a workout on their phone while their Mac is offline. On the Mac, they change the program that produced the workout. Later, both devices see each other’s changes. There is no server in the middle to declare which copy is current. The phone and Mac have to examine the same evidence and reach the same answer independently.

Now add deletion. If the Mac deletes a meal and the phone still has an older copy, the missing file is not enough to communicate what happened. It might have been deleted, or it might simply be unavailable while a storage provider downloads it. Korivo retains a deletion marker so that another device has evidence of the decision. Both applications then have to agree on how long that evidence wins, when it can be removed, and what to do if an old copy reappears.

These examples are why the shared behavior is broader than reading and writing JSON. It includes identity, folder layout, time partitioning, conflict ordering, deletion, duplicate repair, and the sequence in which a device imports and publishes work. It also includes failure behavior. A temporarily unreadable file cannot become an accidental deletion, and an interrupted repair cannot remove the last valid copy of a record.

We already had a substantial system for keeping the two engines aligned. Shared fixtures lived in the central repository and ran against both Swift and TypeScript. They covered the portable record shapes and many of the decisions both applications had to make. This was not a casual arrangement based only on engineers remembering to update both sides.

Even with that system, we found bugs where the implementations or the paths around them diverged. More importantly, the cost grew with every new record and rule. Adding a third platform would require a third engine and a third fixture runner, with every behavioral change implemented and reviewed again. The fixture system helped us manage duplication, but it did not change the fact that the duplication existed.

The migration therefore had four constraints:

1. The portable record format and existing folder layout could not change merely to accommodate the new implementation.
2. The old and new implementations had to coexist long enough to compare their behavior on the same inputs.
3. Read authority, write authority, and durable synchronization state had to move independently so that each change could be rolled back without reverting the whole migration.
4. Platform behavior had to remain native. Rust would not replace SwiftData, Tauri, operating-system file access, application lifecycle management, or either application’s user interface.

The resulting boundary separates shared decisions from platform work. The core decides what a record means, which version wins, whether the view of a folder is complete, and what a synchronization pass needs to do. The applications still perform their own database and filesystem operations. This is important: moving policy into Rust does not make iOS suspension, SwiftData transactions, or desktop file-provider behavior disappear.

The existing fixture system became the specification for the port

A common way to begin a port is to place the old source code on one screen and translate it function by function. We had a better starting point. The central fixture repository already described many results that mattered outside either implementation: the exact bytes for a record, the folder where it belongs, the winner of a conflict, and the effect of a deletion. Swift and TypeScript ran the same examples.

That let Rust target observable behavior rather than the internal structure of either engine. The three implementations could use the natural data structures of their languages. What mattered was that they made the same decision at the boundary.

But a fixture system is only as strong as the path it exercises and the cases it knows it is missing. During the migration we found tests that reached a clean standalone codec while the application used a different path. Some fixture families were thorough but not checked for completeness, so adding a new record kind did not necessarily prove that every relevant suite had enrolled it. In a few cases Swift and TypeScript agreed because they had inherited the same assumption, not because the assumption was correct.

We therefore treated the existing corpus as both specification and code that needed hardening. Full application-path tests were connected to the same codecs the fixtures exercised, and the corpus was expanded to cover behavior that only becomes visible across devices or across more than one synchronization pass.

We compare exact bytes in places where object equality would hide a real difference. Two applications might both decode a timestamp correctly but file it under different days because one uses local time and the other uses UTC. They might both read an absent value as “nothing” but disagree on whether to write the field back as null or leave it out. Either mismatch is enough for two devices to produce different files from the same program.

The fixture suites are divided by responsibility. Wire fixtures cover decode and canonical re-encode. Layout fixtures enumerate record kinds, folders, and partition fields. Decision fixtures cover conflict and deletion policy. Multi-step scenarios cover behavior that is only visible across devices or across more than one reconciliation pass. This division makes a failure attributable: it identifies whether a change affected representation, policy, or orchestration.

We also added completeness checks around the corpus. Every registered sync record kind must have a wire fixture and participate in the relevant layout and deletion tables. Negative fixtures verify that required fields are actually required and that invalid values are rejected by each implementation. These checks prevent an incomplete suite from remaining green when a new record kind or field is added.

This work also exposed product decisions that had previously been buried inside code. Should a deletion on one device beat a later edit made from an older copy on another? For how long? Those are not Rust or JSON questions. They determine what the user sees. We wrote the scenarios in those terms, decided them once, and then required every implementation to reproduce the decision.

One description of the data, without forcing one application architecture

Shared examples tell us when implementations disagree, but they do not remove the duplication that causes disagreement. Swift and TypeScript still had separate declarations for the same program structures. A field added on one side could be missed on the other, or could be given a subtly different optional or numeric type.

We introduced a small neutral schema for the straightforward parts of Korivo’s portable data. From it we generate native Swift and TypeScript types. Rust reads the same record shapes, and the core is exposed to each application through a native boundary. The applications do not share a runtime and do not have to adopt Rust data structures throughout their code.

There was a temptation to make the schema describe everything immediately. We resisted it. Some older records do more than map fields: they accept an old form, normalize it, and write a newer form; others deliberately order their contents before writing. If the schema language cannot express that behavior clearly, generating the code would only hide the exception. Those cases remain handwritten and covered by the same shared examples.

The goal is not to generate the maximum amount of code. It is to have one honest description of the ordinary structure and to keep the genuinely unusual behavior visible.

We moved from small decisions to the full synchronization loop

We started with the parts that can be expressed as simple questions: Can this record be decoded? What are its canonical bytes? Which folder does it belong in? Given two versions, which one wins? These operations can run without opening an application database or touching a user’s sync folder.

That order mattered. When the new implementation disagreed, the cause was confined to a small area. The early work found fields that were being renamed differently, values that survived decoding but were not written back exactly, and unknown data that one implementation preserved while another discarded. We fixed the underlying class of problem and expanded the shared examples rather than teaching Rust to imitate one bad instance.

Once the leaf operations were stable, the next boundary was a wire-shaped snapshot: a sorted view of the records and tombstones observed in one workspace, together with evidence about whether the view is complete. The snapshot deliberately does not expose SwiftData relationships or TypeScript object identity. This lets both applications construct the same core input and prevents merge policy from depending on one platform’s domain model.

After the small decisions matched, we moved up one level. A snapshot gives the core a stable view of what a device observed in a folder. A merge takes the local snapshot, the remote snapshot, and any unpublished local work, then decides what survives. Only after those decisions were independently testable did we build the full reconciliation pass around them.

This bottom-up order made the project slower to start and much faster to debug. A disagreement in a complete sync pass could be traced through observation, decoding, ordering, merging, and writing instead of being treated as one opaque “Rust sync” failure.

The new engine spent a long time being right without being in charge

A fixture corpus covers known behavior. It cannot demonstrate that the corpus contains every shape produced by the applications as they evolve. We therefore integrated the core in stages and initially allowed it to observe development data and full application scenarios without controlling the result.

On the write path, the established implementation continued to prepare and write records. The same logical batch was sent to Rust, and the canonical bytes were compared before any core write capability was enabled. On the read path, complete folder layouts were replayed through both implementations and their snapshots were compared. Divergences were written to a bounded diagnostic ledger. Failure in the comparison path could not block or fail the application save being exercised.

We used separate controls for separate capabilities. Linking the core, invoking a conflict kernel, decoding a batch, mirroring durable state, booting from the mirrored state, preparing writes, and performing writes were not one feature flag. This allowed a proven lower-level operation to remain in service if a later stage had to be disabled.

For each major capability, the migration follows the same progression:

1. Fixture equivalence. The old and new implementations satisfy the same shared cases.
2. Shadow execution. Both process the same application input, but only the established path is authoritative.
3. Passive persistence. The core records a durable mirror that is compared but not used to make product decisions.
4. Read authority. The application can boot from core state while continuing to compare it with the established source.
5. Write preparation and write authority. The core first proposes exact writes for comparison, then performs them behind a separate rollback control.
6. Retirement. The former implementation is removed only after it is no longer needed as a comparator or rollback path.

This progression separates two questions that are easy to blur together. First: can the new engine compute the same answer? Second: can it safely become responsible for that answer while files are unavailable, processes stop, and application state continues changing? Passing the first test does not imply the second.

Once the core could remember, it could begin to reconcile

Synchronization cannot be implemented entirely as pure snapshot functions. The system needs to retain what it has observed, what local work is pending publication, which operation has been acknowledged, and whether a scan was sufficiently complete to justify removing prior state.

The Rust store keeps canonical wire bytes rather than introducing another typed relational representation of the domain. Typed columns would make some local queries easier, but they would create an additional schema and migration surface beside SwiftData, the portable record schema, and Studio’s application models. The core primarily needs stable identity, ordering metadata, canonical payloads, and reconciliation progress, so a wire-byte store is a closer representation of its responsibility.

During the passive stage, application reads and writes remained authoritative while the core received the same records. Comparison was performed over the full logical inventory, not only row counts or successfully decoded records. This distinction led to an explicit completeness marker. If a folder lists ten candidate files and one cannot be read, the result is nine records from an incomplete scan. It is not an authoritative nine-record snapshot. Pruning based on the latter interpretation would convert a temporary download or permission failure into deletion.

Local publication uses an outbox. A change is journaled before it is published to the shared folder, and the exact entry is acknowledged after the write completes. Generations are monotonic so completion of an older write cannot clear a newer edit to the same record. An acknowledgement failure after a successful folder write remains durable health evidence and pending work that can be reconciled; it does not retroactively report an already completed local save as failed.

We found it useful to distinguish three sources of truth during this phase:

• The application database is authoritative for current local intent.
• The shared folder is authoritative for what has been published and can be observed by other devices.
• The core store is authoritative for durable progress between the two.

Confusing them produces subtle failures. Starting a write does not mean a change has been published. Completion of an older write does not acknowledge a newer edit. Seeing fewer files than last time does not prove the missing records were deleted.

The reconciler operates over these distinctions. A pass observes and classifies folder contents, imports remote state, protects pending local work, applies merge and deletion policy, repairs duplicate or mispartitioned files, publishes pending records, and garbage-collects deletion evidence when it is safe to do so. The order is part of the behavior.

Several bugs we found can be explained without knowing anything about the implementation. In one scenario, a newly configured device connected to an existing folder and published its empty starting state before importing the populated state already there. A record edited while an earlier save was still finishing could be incorrectly marked clean when the old save completed. A temporary read failure could make a complete folder appear smaller and cause valid records to be pruned from the mirror.

The fixes follow from the distinctions above. Import before publishing a new device’s defaults. Acknowledge the exact generation that finished, not merely the record that was being written. Treat a scan as incomplete when any expected file cannot be read, and forbid destructive pruning from an incomplete view.

There is no transaction spanning an application database and an arbitrary user-controlled folder. The design therefore favors repeatable operations and recoverable intermediate states. Constructive work precedes destructive cleanup. If a pass stops after relocating a valid copy but before removing a duplicate, the next pass observes redundant data and can finish. Performing those operations in the opposite order can leave no valid copy to recover.

How we tried to prove the tests were testing the dangerous thing

The migration changed how we evaluate tests. Cross-language parity remains necessary, but parity can also reproduce the same defect on every platform. We use several additional forms of evidence.

Negative fixtures demonstrate that malformed records are rejected consistently. Registry checks make it difficult to add a record kind without adding the corresponding contract coverage. Seeded stress tests operate two stores through randomized sequences and compare the result with a smaller oracle. Fault injection interrupts reads, writes, acknowledgements, and repair steps at defined boundaries. Headless end-to-end scenarios run the real application persistence stacks against private data directories and exercise multi-device convergence.

We also check that important tests are sensitive to the fault they claim to cover. If a test is meant to protect a race, we reintroduce the dangerous ordering and confirm that the test fails. This sounds obvious, but it is easy to write a test about concurrent work using a fake that completes immediately. The test passes, the comment looks reassuring, and the race never occurred.

The dual implementations found defects in the established paths as well as in Rust. Among them were publishing empty state before importing the active folder, clearing a dirty record after a newer edit arrived, deriving canonical key order from one exact object shape, pruning a mirror after a partial scan, and repairing duplicates in an order that could remove the only good copy. These were fixed in the established paths before the corresponding responsibility moved into the core. The migration has therefore improved the system throughout development rather than deferring all benefit until final cutover.

What this approach costs, and where the migration stands

This approach temporarily increases complexity. During the migration we maintain the established engines, a new core, platform adapters, generated bindings, comparison paths, feature controls, diagnostic ledgers, and recovery behavior. A direct replacement would produce a cleaner source tree sooner. It would also remove the ability to compare both implementations on the same application-generated input and would make rollback an application-wide event.

Rust adds another toolchain and a boundary between each application and the core. Calls that cross that boundary require better instrumentation than ordinary local functions. A shared core also does not eliminate platform-specific failure modes. Phones still suspend work, local databases still have transaction boundaries, and desktop storage providers still make files temporarily unavailable.

The primary benefit is not execution speed. It is having fewer versions of the truth. A new platform should not need another interpretation of how records are written, which edit wins, where files belong, or how deletions survive an offline device. It should need a native application around an established policy engine.

The core currently implements typed wire handling, canonical representation, conflict decisions, folder snapshots, durable mirror and outbox state, snapshot merge, and the reconciliation pass behind controlled rollout boundaries. Swift uses generated bindings for core policy, and Studio reaches the same library through Tauri. Some older Swift and TypeScript behavior remains in place as a comparison oracle or rollback path.

That duplication will be removed after the relevant core path has passed fixture equivalence, application-path comparison, failure testing, and a period of authoritative operation. Until then, the additional implementation is part of the verification system. The migration is complete when the core is the ordinary source of cross-platform policy and the transitional comparison infrastructure is no longer needed.