Live data from Hacker News

LiteFS a FUSE-based file system for replicating SQLite

github.com

41–50 of 66 posts

Re: LiteFS a FUSE-based file system for replicating SQLite

#41

Earlier quoted context omitted.

Thanks! I think sharding is really interesting -- especially with a lightweight database like SQLite. I'm not looking for contributions right now but I would love to hear any feedback on the approach taken with LiteFS. I want to make it as easy to run as possible.

I'd give LiteFS a run soon. SqlLite is cross-platform, but seems like LiteFS is not. True? https://github.com/superfly/litefs

Yes, that's correct. It's Linux only right now although I'll probably support macOS too (via MacFUSE).

Re: LiteFS a FUSE-based file system for replicating SQLite

#42

LiteFS author here (also Litestream author). I'm happy to answer any questions folks have about how it works or what's on the roadmap.

A while ago, when I discovered litestream, I played around with it and just setup cloning to a local file://; is this the same idea?

Sorta. LiteFS sets up a FUSE mount so once you have that in place then you can create a database within that mount directory and it is automatically replicated.

Re: LiteFS a FUSE-based file system for replicating SQLite

#43
And I /just/ got my infrastructure bits and pieces running Litestream! Guess I'll have to figure out if it's worth switching to this -- my gut reaction is no, since I only really run one pod at a time, so Litestream serves the purpose of not only saving the database offsite but also restoring it. But I will be keeping a very close eye on this thanks in part to my love of SQLite.

Hats off to Ben and Fly.io, you're doing some cool stuff.

Re: LiteFS a FUSE-based file system for replicating SQLite

#44

Tangentially related: I'd like to use litestream but my SQLite files are several gigabytes, is there a way to lazily download the db only once it's being accessed? (using something like userfaultfd maybe? just thinking out loud)

> is there a way to lazily download the db only once it's being accessed?

Not sure what's your full scenario, because you mentioned "lazily download" so thought you might have a luck here https://news.ycombinator.com/item?id=27016630

tl;dr: using http range and a smaller page size - might be the way to go

Re: LiteFS a FUSE-based file system for replicating SQLite

#45
post #43

And I /just/ got my infrastructure bits and pieces running Litestream! Guess I'll have to figure out if it's worth switching to this -- my gut reaction is no, since I only really run one pod at a time, so Litestream serves the purpose of not only saving the database offsite but also restoring it. But I will be keeping a very close eye on this thanks in part to my love of SQLite. Hats off to Ben and Fly.io, you're doi…

Thanks! Yeah, if you don't need multiple replicas then Litestream should work just fine. I'd say stick to that for now.

Re: LiteFS a FUSE-based file system for replicating SQLite

#46
post #30
post #13

I have an adjacent problem, and I haven't been able to find anyone who has a fix for me. One perfectly reasonable use case for a read replica of a database is a bastion server. Database + web server on a machine that is firewalled both from the internet and from the business network. With read only access there is a much smaller blast radius if someone manages to compromise the machine. The problem is that every sing…

Look into reverse SSH tunnelling. SSH from primary to secondary, which then connects back to the primary through the already-established SSH connection.

That's the closest I've been able to come up with, but it does have the problem that anything local can typically connect to that tunnel. In the bastion situation we generally don't assume that the machine is not compromised. Otherwise why did we put it outside of the firewall?

To be fair, there are a number of ways a hostile endpoint can screw with another server even just by screwing around with TCP protocol behavior, so perhaps I'm putting too fine a point on it.

Re: LiteFS a FUSE-based file system for replicating SQLite

#48
> LiteFS is intended to provide easy, live, asychronous replication across ephemeral nodes in a cluster. This approach makes trade-offs as compared with simpler disaster recovery tools such as Litestream and more complex but strongly-consistent tools such as rqlite.

I think rqlite having a single binary that handles Raft / consensus _and_ includes SQLITE makes it simpler. Beyond 'hello world', Consul isn't trivial to run and Fly have blogged about this [0]

0. https://fly.io/blog/a-foolish-consistency/

Re: LiteFS a FUSE-based file system for replicating SQLite

#49

LiteFS author here (also Litestream author). I'm happy to answer any questions folks have about how it works or what's on the roadmap.

Thanks for sharing this!

As WAL'd sqlite on networked filesystems is a no-go, once LiteFS gets wal support I think this will be a savior software for many operators with existing sqlite deployments.

Could we even host LiteFS with underlying dbs on a filesystem that otherwise wouldn't play well with WAL (say nfs, ceph or gluster)? An effectively single-node deployment that would achieve redundancy through the FS.

Re: LiteFS a FUSE-based file system for replicating SQLite

#50

> LiteFS is intended to provide easy, live, asychronous replication across ephemeral nodes in a cluster. This approach makes trade-offs as compared with simpler disaster recovery tools such as Litestream and more complex but strongly-consistent tools such as rqlite. I think rqlite having a single binary that handles Raft / consensus _and_ includes SQLITE makes it simpler. Beyond 'hello world', Consul isn't trivial to…

rqlite is very cool, but it's also much more ambitious than LiteFS; it's Raft consistency for every instance of the database, where Litestream/LiteFS is replication (for single-writer multi-reader setups, where reads are answered quickly from edges or read caches, and writes are funneled to a central node --- with LiteFS, an elected central node). Raft is, of course, more powerful, but it's also its own whole thing to manage and monitor.

The advantage of LiteFS/Litestream is that, for the most part, the database is "just" SQLite. You can't really say that, to the same extent, about rqlite.

I hope rqlite takes off! It's a good project.

We've spent a lot of time at Fly.io wrestling with Consul, but that's because we abuse it. That's what the article is about: we shoehorned Consul into a part of our architecture where we're taxed for features it has that we don't actually use (the overwhelming majority of all the data we have in Consul is stuff for which there's a single, continually available source of truth for the data, and Consul was just a "convenient" way to replicate it). Consul is great for the stuff it's meant for.

I wouldn't hesitate to reach for Consul in a new design. I just wouldn't use it for the thing we used it for.

Post reply on HN