SQLite database sync between devices: safe approaches
Solved Email & Outlook
AJ
Andrew Jackson
June 22, 2021
2 replies
9,680 views
Reviewed by moderators

My application stores its data in SQLite and users now want the same data on their laptop and desktop. First instinct was putting the database file in Dropbox.

Something tells me that instinct is dangerous. What are the safe architectures for syncing SQLite?

Accepted Answer
Verified by Kerry Morris, Forum Moderator ยท Reviewed June 2021

Trust the something, the Dropbox instinct corrupts databases and the mechanism is worth thirty seconds so the danger stays vivid: SQLite writes involve the main file plus a journal or WAL file in coordination, and file synchronizers copy files independently mid write, delivering combinations to the other machine that never existed on the first. The database that arrives can be subtly wrong rather than obviously broken, the worse outcome. The safe architectures, ascending by effort:

Closed file shuttling, the honest minimum: sync only when the application has fully closed the database, checkpointed the WAL and released everything, treating the file as a document passed back and forth. Works genuinely for one user moving between machines at different times, enforced by the application refusing to open a database it did not cleanly close or by syncing an exported copy rather than the live file. The moment both machines run the application simultaneously this architecture ends, no negotiation.

Change log and merge, sync as a feature: the application records its own changes in a log table, machines exchange logs through any transport including that same Dropbox folder, safely now since log files are append only documents and each machine applies the other's changes with conflict rules the application defines. This is real engineering effort, unique row identifiers, tombstones for deletes, last write wins or better per table, but it delivers true multi device sync while keeping SQLite's zero administration character, and it is the architecture behind most polished offline first applications.

Promotion to client server, the boundary recognition: sometimes the users asking for multi device are announcing the application outgrew embedded storage, and a server holding the database, whether a hosted Postgres or SQLite itself behind a small API, dissolves the sync problem into the ordinary client server model. The replication tools in the SQLite ecosystem, Litestream style continuous replication to object storage, solve the related backup problem elegantly but are one directional, disaster recovery rather than multi device sync, worth knowing to avoid conscripting them into the wrong job.

For a typical desktop application the change log architecture is usually the destination, with closed file shuttling as the shippable interim and the file in Dropbox instinct redirected into syncing the change logs where it becomes safe by design.

Shipped the closed file mode this week as the interim with a lock refusing simultaneous opens, and the change log design is on the roadmap with your conflict rule notes attached. The corruption mechanism explanation went straight into the team wiki, vivid indeed.