LiteFS a FUSE-based file system for replicating SQLite • Hacker News Board

Re: LiteFS a FUSE-based file system for replicating SQLite

#21

Post by ctur » Tue, Jul 26, 2022, 7:54 PM UTC

If a "FUSE to replicate SQLite" solution came from anywhere else, I'd be quite skeptical, but there is a lot of very interesting tech coming out of fly.io these days and Ben certainly knows this space well. It still feels a little like a hack and piercing of layers of abstraction (less so than, say, litestream).

I love it when at first glance it isn't clear if a project is a crazy idea from someone just goofing around vs a highly leveraged crazy idea that will be a foundational part of a major technology shift.

I suspect it's the latter and the strategy though is to layer this on top of litestream to create an easy way to use sqlite transparently in a widely distributed multi-node environment (litestream providing the backups and/or readonly replication to remote sites, with LiteFS handling low latency local access in a cluster, POP, or data center).

Cool stuff. It will be fun to see where fly takes this :)

Re: LiteFS a FUSE-based file system for replicating SQLite

#22

Post by benbjohnson » Tue, Jul 26, 2022, 7:56 PM UTC

anon291 wrote: post #16

benbjohnson wrote: post #2

LiteFS author here (also Litestream author). I'm happy to answer any questions folks have about how it works or what's on the roadmap.

Why a fuse filesystem instead of just using sqlite's vfs features? FUSE adds an extra user-kernel-user context switch which is expensive compared to a quick user-user call for the vfs drivers.

The VFS system is great and I plan to support that as well. SQLite is simple to get started with but it can be complicated to integrate a VFS depending on the language using. My goal is to make the deployment as simple as possible and also to support legacy applications.

There's some interesting work I'm looking at with the VFS and WASM where you could deploy something like this across pure serverless functions like Vercel or Deno.

Re: LiteFS a FUSE-based file system for replicating SQLite

#23

Post by benbjohnson » Tue, Jul 26, 2022, 7:59 PM UTC

liuliu wrote: post #17

benbjohnson wrote: post #2

LiteFS author here (also Litestream author). I'm happy to answer any questions folks have about how it works or what's on the roadmap.

Does LiteFS needs to do "pattern matching" to know what SQLite writes (a txn or not e.g.)? With Litestream, it seems simply use SQLite API. Just curious how do you think through this.

Transactions start with the creation of the journal file and end when it's unlinked so there's some "pattern matching" but it's not terribly complicated. Once the file is unlinked, LiteFS can read and verify the journal and pull the change set of pages from the database file.

Litestream interacts with the SQLite API for locking but it uses a regular file descriptor to read the WAL. It has to do similar parsing of pages but it's a bit more complicated since it needs to re-verify its position every time it reads from the WAL.

Re: LiteFS a FUSE-based file system for replicating SQLite

#24

Post by tptacek » Tue, Jul 26, 2022, 8:03 PM UTC

benbjohnson wrote: post #22

anon291 wrote: post #16

Earlier quoted context omitted.

Why a fuse filesystem instead of just using sqlite's vfs features? FUSE adds an extra user-kernel-user context switch which is expensive compared to a quick user-user call for the vfs drivers.

The VFS system is great and I plan to support that as well. SQLite is simple to get started with but it can be complicated to integrate a VFS depending on the language using. My goal is to make the deployment as simple as possible and also to support legacy applications. There's some interesting work I'm looking at with the VFS and WASM where you could deploy something like this across pure serverless functions like…

Would doing the VFS thing require participating applications to be linked to a LiteFS library?

Re: LiteFS a FUSE-based file system for replicating SQLite

#25

Post by tmp_anon_22 » Tue, Jul 26, 2022, 8:03 PM UTC

ctur wrote: post #21

If a "FUSE to replicate SQLite" solution came from anywhere else, I'd be quite skeptical, but there is a lot of very interesting tech coming out of fly.io these days and Ben certainly knows this space well. It still feels a little like a hack and piercing of layers of abstraction (less so than, say, litestream). I love it when at first glance it isn't clear if a project is a crazy idea from someone just goofing aroun…

> a highly leveraged crazy idea that will be a foundational part of a major technology shift

Has anything other then the Cloud presented a true foundational shift in how applications are built? Kubernetes, Serverless, Blockchain, React, Swift, these things are big but not big enough.

I think we just like pretending every little thing is the next big thing.

Re: LiteFS a FUSE-based file system for replicating SQLite

#26

Post by hobo_mark » Tue, Jul 26, 2022, 8:04 PM UTC

Tangentially related: I'd like to use litestream but my SQLite files are several gigabytes, is there a way to lazily download the db only once it's being accessed? (using something like userfaultfd maybe? just thinking out loud)

Re: LiteFS a FUSE-based file system for replicating SQLite

#27

Post by benbjohnson » Tue, Jul 26, 2022, 8:09 PM UTC

ctur wrote: post #21

If a "FUSE to replicate SQLite" solution came from anywhere else, I'd be quite skeptical, but there is a lot of very interesting tech coming out of fly.io these days and Ben certainly knows this space well. It still feels a little like a hack and piercing of layers of abstraction (less so than, say, litestream). I love it when at first glance it isn't clear if a project is a crazy idea from someone just goofing aroun…

Thanks for the vote of confidence! I can understand the "hack" feel -- it's a trade-off. If I wrote it the "proper" way and integrated directly into the SQLite source or used a VFS then it'd be a lot harder to deploy for most folks. By making it a FUSE file system, someone can use it without really knowing much about it from the application's perspective.

As for strategy, it unfortunately doesn't work to layer with Litestream as backups need some strict control over who is the current primary. Instead, I'm adding S3 replication support [1] directly into LiteFS. LiteFS also uses a different transactional file format called LTX so it wouldn't be compatible with Litestream. The LTX format is optimized for compactions so point-in-time restores can be nearly instant.

The end goal isn't much of a secret. We want to let folks spin up nodes in regions across the world, automatically connect to one another, and have the whole thing have the ease of a single node app. We still have a ways to go on that vision but that's what we're going for.

[1] https://github.com/superfly/litefs/issues/18

Re: LiteFS a FUSE-based file system for replicating SQLite

#28

Post by 0x457 » Tue, Jul 26, 2022, 8:11 PM UTC

mandeepj wrote: post #14

hinkley wrote: post #13

I have an adjacent problem, and I haven't been able to find anyone who has a fix for me. One perfectly reasonable use case for a read replica of a database is a bastion server. Database + web server on a machine that is firewalled both from the internet and from the business network. With read only access there is a much smaller blast radius if someone manages to compromise the machine. The problem is that every sing…

> a database without multi-master support I believe Cassandra does not have a Master\follower architecture; it's following a ring based structure.

Yeah, Cassandra uses a Distributed Hash Table with Consistent Hashing.

When a new node added, it takes responsibility for a segment of a ring. When node removed, it's segment get redistributed.

It's however in no way a drop-in replacement for RDBMS and requires a careful planning around application read and writes patterns that is very different from your typical RDBMS. Definitely, can't be used in this scenario - every node needs to be able to access every other node and client must be able to access at least one node.

Re: LiteFS a FUSE-based file system for replicating SQLite

#29

Post by benbjohnson » Tue, Jul 26, 2022, 8:11 PM UTC

hobo_mark wrote: post #26

Tangentially related: I'd like to use litestream but my SQLite files are several gigabytes, is there a way to lazily download the db only once it's being accessed? (using something like userfaultfd maybe? just thinking out loud)

I'm not sure how that could be implemented with Litestream since it runs as a separate process. It could be possible with LiteFS where it just pages in data on the fly. That's on the roadmap but it's probably still a ways out.

Re: LiteFS a FUSE-based file system for replicating SQLite

#30

Post by rhizome » Tue, Jul 26, 2022, 8:12 PM UTC

hinkley wrote: post #13

I have an adjacent problem, and I haven't been able to find anyone who has a fix for me. One perfectly reasonable use case for a read replica of a database is a bastion server. Database + web server on a machine that is firewalled both from the internet and from the business network. With read only access there is a much smaller blast radius if someone manages to compromise the machine. The problem is that every sing…

Look into reverse SSH tunnelling. SSH from primary to secondary, which then connects back to the primary through the already-established SSH connection.