Live data from Hacker News

LiteFS a FUSE-based file system for replicating SQLite

github.com

51–60 of 66 posts

Re: LiteFS a FUSE-based file system for replicating SQLite

#51
post #49

LiteFS author here (also Litestream author). I'm happy to answer any questions folks have about how it works or what's on the roadmap.

Thanks for sharing this! As WAL'd sqlite on networked filesystems is a no-go, once LiteFS gets wal support I think this will be a savior software for many operators with existing sqlite deployments. Could we even host LiteFS with underlying dbs on a filesystem that otherwise wouldn't play well with WAL (say nfs, ceph or gluster)? An effectively single-node deployment that would achieve redundancy through the FS.

That would be an amazing hack. "LiteFS makes sqlite work on NFS".

Re: LiteFS a FUSE-based file system for replicating SQLite

#52
Having been a happy user developing solutions around SQLite for a good amount of time, I find all these "enterprisy" hacks / extension curious.

There are great solutions out there that handle these things and have for a long time.

I know SQLite has become the new hotness, but I really do not want SQLite to get good at all these things because then it would no longer be great at what it does marvelously already.

Re: LiteFS a FUSE-based file system for replicating SQLite

#53

Having been a happy user developing solutions around SQLite for a good amount of time, I find all these "enterprisy" hacks / extension curious. There are great solutions out there that handle these things and have for a long time. I know SQLite has become the new hotness, but I really do not want SQLite to get good at all these things because then it would no longer be great at what it does marvelously already.

This is about as non-enterprisey as it gets. It's built to make sqlite work better for tiny little node.js apps running on very cheap hosting.

Re: LiteFS a FUSE-based file system for replicating SQLite

#54

Tangentially related: I'd like to use litestream but my SQLite files are several gigabytes, is there a way to lazily download the db only once it's being accessed? (using something like userfaultfd maybe? just thinking out loud)

Verneuil (https://github.com/backtrace-labs/verneuil) offers that for read replicas, because backtrace as well has some multi-GB sqlite DBs. It's a VFS (loadable at runtime as a sqlite extension), instead of a filesystem.

I don't remember if that's in the docs; commit that adds the configuration field https://github.com/backtrace-labs/verneuil/commit/027318ba74... and commit for the enum https://github.com/backtrace-labs/verneuil/blob/e6697498f3ba...

(the actual implementation is nothing special https://github.com/backtrace-labs/verneuil/commit/b6bdfcf7bc...)

Issue for the feature: https://github.com/backtrace-labs/verneuil/issues/12

Re: LiteFS a FUSE-based file system for replicating SQLite

#55
post #16

Earlier quoted context omitted.

Why a fuse filesystem instead of just using sqlite's vfs features? FUSE adds an extra user-kernel-user context switch which is expensive compared to a quick user-user call for the vfs drivers.

The VFS system is great and I plan to support that as well. SQLite is simple to get started with but it can be complicated to integrate a VFS depending on the language using. My goal is to make the deployment as simple as possible and also to support legacy applications. There's some interesting work I'm looking at with the VFS and WASM where you could deploy something like this across pure serverless functions like…

It just so happens that I am working on a versioned file system for sqlite based on its VFS interface. Planning to release it in a week's time. It will be in pre-alpha stage and its more of a proof-of-concept.

Having worked with FUSE in the past, there is always some performance penalty, but the advantage is that there is no change in the application.

The VFS will need change in the application but less overhead. Trade-offs!

Re: LiteFS a FUSE-based file system for replicating SQLite

#56
post #24

Earlier quoted context omitted.

Would doing the VFS thing require participating applications to be linked to a LiteFS library?

That's a good question. I thought that was the case but I just double checked and it looks like you can load a VFS as an extension [1]. [1] https://www.sqlite.org/loadext.html

I can confirm that it does. I am working on it as I write this :).

Technically, an extension and VFS are separate things.

An extension when loads, allows you to register the VFS with the Sqlite. It can also register virtual tables as well.

Re: LiteFS a FUSE-based file system for replicating SQLite

#57
post #6

Earlier quoted context omitted.

Given a situation where replication is desirable, is using SQLite + LiteFS a better choice than just replicated Postgres?

Postgres is great and replicating it can work as well. One benefit to SQLite is that it's in-process so you avoid most per-query latency so you don't have to worry as much about N+1 query performance issues. From my benchmarks, I see latency from an application node to a Postgres node as high as 1 millisecond -- even if both are in the same region. SQLite, on the other hand, has per-query latency overhead of about 10…

SQLite v Postgres here is apples v oranges. Postgres is multi-reader multi-writer whereas SQLite is single-writer multi-reader among other things. They are both very fine databases but solve different use cases.

Re: LiteFS a FUSE-based file system for replicating SQLite

#58

Earlier quoted context omitted.

Postgres is great and replicating it can work as well. One benefit to SQLite is that it's in-process so you avoid most per-query latency so you don't have to worry as much about N+1 query performance issues. From my benchmarks, I see latency from an application node to a Postgres node as high as 1 millisecond -- even if both are in the same region. SQLite, on the other hand, has per-query latency overhead of about 10…

SQLite v Postgres here is apples v oranges. Postgres is multi-reader multi-writer whereas SQLite is single-writer multi-reader among other things. They are both very fine databases but solve different use cases.

From https://www.sqlite.org/isolation.html :

> Isolation And Concurrency: SQLite implements isolation and concurrency control (and atomicity) using transient journal files that appear in the same directory as the database file. There are two major "journal modes". The older "rollback mode" corresponds to using the "DELETE", "PERSIST", or "TRUNCATE" options to the journal_mode pragma. In rollback mode, changes are written directly into the database file, while simultaneously a separate rollback journal file is constructed that is able to restore the database to its original state if the transaction rolls back. Rollback mode (specifically DELETE mode, meaning that the rollback journal is deleted from disk at the conclusion of each transaction) is the current default behavior.

> Since version 3.7.0 (2010-07-21), SQLite also supports "WAL mode". In WAL mode, changes are not written to the original database file. Instead, changes go into a separate "write-ahead log" or "WAL" file. Later, after the transaction commits, those changes will be moved from the WAL file back into the original database in an operation called "checkpoint". WAL mode is enabled by running "PRAGMA journal_mode=WAL".

> In rollback mode, SQLite implements isolation by locking the database file and preventing any reads by other database connections while each write transaction is underway. Readers can be active at the beginning of a write, before any content is flushed to disk and while all changes are still held in the writer's private memory space. But before any changes are made to the database file on disk, all readers must be (temporarily) expelled in order to give the writer exclusive access to the database file. Hence, readers are prohibited from seeing incomplete transactions by virtue of being locked out of the database while the transaction is being written to disk. Only after the transaction is completely written and synced to disk and committed are the readers allowed back into the database. Hence readers never get a chance to see partially written changes.

> WAL mode permits simultaneous readers and writers. It can do this because changes do not overwrite the original database file, but rather go into the separate write-ahead log file. That means that readers can continue to read the old, original, unaltered content from the original database file at the same time that the writer is appending to the write-ahead log. In WAL mode, SQLite exhibits "snapshot isolation". When a read transaction starts, that reader continues to see an unchanging "snapshot" of the database file as it existed at the moment in time when the read transaction started. Any write transactions that commit while the read transaction is active are still invisible to the read transaction, because the reader is seeing a snapshot of database file from a prior moment in time.

> An example: Suppose there are two database connections X and Y. X starts a read transaction using BEGIN followed by one or more SELECT statements. Then Y comes along and runs an UPDATE statement to modify the database. X can subsequently do a SELECT against the records that Y modified but X will see the older unmodified entries because Y's changes are all invisible to X while X is holding a read transaction. If X wants to see the changes that Y made, then X must end its read transaction and start a new one (by running COMMIT followed by another BEGIN.)

Or:

  ROLLBACK; // cancel the tx e.g.  because a different dbconn thread detected updated data before the tx was to be COMMITted.

  // Replay the tx 
  BEGIN;
  // replay the same SQL statements
  COMMIT;

Re: LiteFS a FUSE-based file system for replicating SQLite

#59

Earlier quoted context omitted.

SQLite v Postgres here is apples v oranges. Postgres is multi-reader multi-writer whereas SQLite is single-writer multi-reader among other things. They are both very fine databases but solve different use cases.

From https://www.sqlite.org/isolation.html : > Isolation And Concurrency: SQLite implements isolation and concurrency control (and atomicity) using transient journal files that appear in the same directory as the database file. There are two major "journal modes". The older "rollback mode" corresponds to using the "DELETE", "PERSIST", or "TRUNCATE" options to the journal_mode pragma. In rollback mode, changes are wri…

Shit. I was wrong. Thank you for sharing.

Re: LiteFS a FUSE-based file system for replicating SQLite

#60
post #34

Earlier quoted context omitted.

Transactions start with the creation of the journal file and end when it's unlinked so there's some "pattern matching" but it's not terribly complicated. Once the file is unlinked, LiteFS can read and verify the journal and pull the change set of pages from the database file. Litestream interacts with the SQLite API for locking but it uses a regular file descriptor to read the WAL. It has to do similar parsing of pag…

I am under the impression that if you are in the WAL mode, there is no rollback journal created?

Yes, there is no rollback journal in WAL mode, but the WAL is its own file.
Post reply on HN