LiteFS a FUSE-based file system for replicating SQLite
1–10 of 66 posts
Re: LiteFS a FUSE-based file system for replicating SQLite
#2LiteFS author here (also Litestream author). I'm happy to answer any questions folks have about how it works or what's on the roadmap.
Re: LiteFS a FUSE-based file system for replicating SQLite
#3LiteFS author here (also Litestream author). I'm happy to answer any questions folks have about how it works or what's on the roadmap.
Kudos on the release! I've done almost exactly the same thing in Rust with WAL only; writes are appended to a distributed wal which achieves consensus by raft and distributed time leases. it all worked so nearly I was impressed. It was for a client though, so it's unfortunately proprietary.
Re: LiteFS a FUSE-based file system for replicating SQLite
#4LiteFS author here (also Litestream author). I'm happy to answer any questions folks have about how it works or what's on the roadmap.
Awesome! Glad to see it.
I was very excited for the experimental support for live replication in Litestream, but as I understand that didn't pan out. This looks to be the successor, with fanout replication an explicit feature.
Using a FUSE layer that's detecting changes is sure to have some performance tradeoffs. What benchmarks are under way? Do you need any help?
Re: LiteFS a FUSE-based file system for replicating SQLite
#5LiteFS author here (also Litestream author). I'm happy to answer any questions folks have about how it works or what's on the roadmap.
Kudos on the release! I've done almost exactly the same thing in Rust with WAL only; writes are appended to a distributed wal which achieves consensus by raft and distributed time leases. it all worked so nearly I was impressed. It was for a client though, so it's unfortunately proprietary.
Thanks! That sounds like a fun project. Did you run the WAL writes through Raft or did you just use Raft for leader election?
Re: LiteFS a FUSE-based file system for replicating SQLite
#6LiteFS author here (also Litestream author). I'm happy to answer any questions folks have about how it works or what's on the roadmap.
Given a situation where replication is desirable, is using SQLite + LiteFS a better choice than just replicated Postgres?
Re: LiteFS a FUSE-based file system for replicating SQLite
#7LiteFS author here (also Litestream author). I'm happy to answer any questions folks have about how it works or what's on the roadmap.
Given a situation where replication is desirable, is using SQLite + LiteFS a better choice than just replicated Postgres?
Postgres is great and replicating it can work as well. One benefit to SQLite is that it's in-process so you avoid most per-query latency so you don't have to worry as much about N+1 query performance issues. From my benchmarks, I see latency from an application node to a Postgres node as high as 1 millisecond -- even if both are in the same region. SQLite, on the other hand, has per-query latency overhead of about 10-20µs.
Another goal is to simplify deployment. It's a lot easier and cost-effective to just deploy out 10 application nodes across a bunch of regions rather than having to also deploy Postgres replicas in each of those regions too.
Re: LiteFS a FUSE-based file system for replicating SQLite
#8LiteFS author here (also Litestream author). I'm happy to answer any questions folks have about how it works or what's on the roadmap.
Awesome! Glad to see it. I was very excited for the experimental support for live replication in Litestream, but as I understand that didn't pan out. This looks to be the successor, with fanout replication an explicit feature. Using a FUSE layer that's detecting changes is sure to have some performance tradeoffs. What benchmarks are under way? Do you need any help?
Thanks! In hindsight, I'm glad I pulled the live replication in Litestream and left it focused on disaster recovery only. A lot of what I'm doing in LiteFS would be much more difficult if I didn't have control over the write path. I'm able to perform rolling checksums to verify database integrity across replicas and that would be nearly impossible with Litestream (which runs as a separate process).
The FUSE layer does have performance trade-offs. Read queries aren't affected as much since the OS page cache is in the kernel and avoids the FUSE layer entirely. Most databases are under 10GB (or honestly probably less than 1GB) so you can fit most of that in the page cache for any moderately sized VM. The write side will incur a performance hit but I haven't taken benchmarks yet. There's a lot that can still be optimized around that.
The biggest help that I could use is just to get feedback on how it feels to use and any ways I can improve it. The end goal is to make it ridiculously simple to spin up and "just work" so any ideas towards that goal would be awesome.
Re: LiteFS a FUSE-based file system for replicating SQLite
#9LiteFS author here (also Litestream author). I'm happy to answer any questions folks have about how it works or what's on the roadmap.
Very cool. I've been working on CRDTs for SQLite to allow for conflict-free replication. Its still super early but I have a prototype that works and, if you're up to it, I would love to pick your brain on the details of sqlite WALs and journaling.
Re: LiteFS a FUSE-based file system for replicating SQLite
#10LiteFS author here (also Litestream author). I'm happy to answer any questions folks have about how it works or what's on the roadmap.
Very cool. I've been working on CRDTs for SQLite to allow for conflict-free replication. Its still super early but I have a prototype that works and, if you're up to it, I would love to pick your brain on the details of sqlite WALs and journaling. https://www.loom.com/share/0934f93364d340e0ba658146a974edb4
Right on, I always thought a CRDT SQLite database would be awesome. I thought about looking at the session extension[1] to handle the merges but I didn't get very far into it.
Feel free to ping me at benbjohnson@yahoo.com if you want to chat WAL & journaling. I think it'd be difficult to implement CRDT at the journal level since it's all physical pages. Merging data in pages is going to sometimes cause splits and that'll be hairy to deal with.