Earlier quoted context omitted.
Thanks! I think sharding is really interesting -- especially with a lightweight database like SQLite. I'm not looking for contributions right now but I would love to hear any feedback on the approach taken with LiteFS. I want to make it as easy to run as possible.
I'd give LiteFS a run soon. SqlLite is cross-platform, but seems like LiteFS is not. True? https://github.com/superfly/litefs
LiteFS a FUSE-based file system for replicating SQLite
41–50 of 66 posts
Re: LiteFS a FUSE-based file system for replicating SQLite
#42LiteFS author here (also Litestream author). I'm happy to answer any questions folks have about how it works or what's on the roadmap.
A while ago, when I discovered litestream, I played around with it and just setup cloning to a local file://; is this the same idea?
Re: LiteFS a FUSE-based file system for replicating SQLite
#43Hats off to Ben and Fly.io, you're doing some cool stuff.
Re: LiteFS a FUSE-based file system for replicating SQLite
#44Tangentially related: I'd like to use litestream but my SQLite files are several gigabytes, is there a way to lazily download the db only once it's being accessed? (using something like userfaultfd maybe? just thinking out loud)
Not sure what's your full scenario, because you mentioned "lazily download" so thought you might have a luck here https://news.ycombinator.com/item?id=27016630
tl;dr: using http range and a smaller page size - might be the way to go
Re: LiteFS a FUSE-based file system for replicating SQLite
#45And I /just/ got my infrastructure bits and pieces running Litestream! Guess I'll have to figure out if it's worth switching to this -- my gut reaction is no, since I only really run one pod at a time, so Litestream serves the purpose of not only saving the database offsite but also restoring it. But I will be keeping a very close eye on this thanks in part to my love of SQLite. Hats off to Ben and Fly.io, you're doi…
Re: LiteFS a FUSE-based file system for replicating SQLite
#46I have an adjacent problem, and I haven't been able to find anyone who has a fix for me. One perfectly reasonable use case for a read replica of a database is a bastion server. Database + web server on a machine that is firewalled both from the internet and from the business network. With read only access there is a much smaller blast radius if someone manages to compromise the machine. The problem is that every sing…
Look into reverse SSH tunnelling. SSH from primary to secondary, which then connects back to the primary through the already-established SSH connection.
To be fair, there are a number of ways a hostile endpoint can screw with another server even just by screwing around with TCP protocol behavior, so perhaps I'm putting too fine a point on it.
Re: LiteFS a FUSE-based file system for replicating SQLite
#47Re: LiteFS a FUSE-based file system for replicating SQLite
#48I think rqlite having a single binary that handles Raft / consensus _and_ includes SQLITE makes it simpler. Beyond 'hello world', Consul isn't trivial to run and Fly have blogged about this [0]
Re: LiteFS a FUSE-based file system for replicating SQLite
#49LiteFS author here (also Litestream author). I'm happy to answer any questions folks have about how it works or what's on the roadmap.
As WAL'd sqlite on networked filesystems is a no-go, once LiteFS gets wal support I think this will be a savior software for many operators with existing sqlite deployments.
Could we even host LiteFS with underlying dbs on a filesystem that otherwise wouldn't play well with WAL (say nfs, ceph or gluster)? An effectively single-node deployment that would achieve redundancy through the FS.
Re: LiteFS a FUSE-based file system for replicating SQLite
#50> LiteFS is intended to provide easy, live, asychronous replication across ephemeral nodes in a cluster. This approach makes trade-offs as compared with simpler disaster recovery tools such as Litestream and more complex but strongly-consistent tools such as rqlite. I think rqlite having a single binary that handles Raft / consensus _and_ includes SQLITE makes it simpler. Beyond 'hello world', Consul isn't trivial to…
The advantage of LiteFS/Litestream is that, for the most part, the database is "just" SQLite. You can't really say that, to the same extent, about rqlite.
I hope rqlite takes off! It's a good project.
We've spent a lot of time at Fly.io wrestling with Consul, but that's because we abuse it. That's what the article is about: we shoehorned Consul into a part of our architecture where we're taxed for features it has that we don't actually use (the overwhelming majority of all the data we have in Consul is stuff for which there's a single, continually available source of truth for the data, and Consul was just a "convenient" way to replicate it). Consul is great for the stuff it's meant for.
I wouldn't hesitate to reach for Consul in a new design. I just wouldn't use it for the thing we used it for.