Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LiteFS a FUSE-based file system for replicating SQLite

github.com

Post reply on HN
Print view View original

31–40 of 66 posts

Re: LiteFS a FUSE-based file system for replicating SQLite

#31

Post by benbjohnson »

tptacek wrote: post #24
benbjohnson wrote: post #22

Earlier quoted context omitted.

The VFS system is great and I plan to support that as well. SQLite is simple to get started with but it can be complicated to integrate a VFS depending on the language using. My goal is to make the deployment as simple as possible and also to support legacy applications. There's some interesting work I'm looking at with the VFS and WASM where you could deploy something like this across pure serverless functions like…

Would doing the VFS thing require participating applications to be linked to a LiteFS library?

That's a good question. I thought that was the case but I just double checked and it looks like you can load a VFS as an extension [1].

[1] https://www.sqlite.org/loadext.html

Re: LiteFS a FUSE-based file system for replicating SQLite

#32

Post by dan353hehe »

benbjohnson wrote: post #8
GauntletWizard wrote: post #4

Earlier quoted context omitted.

Awesome! Glad to see it. I was very excited for the experimental support for live replication in Litestream, but as I understand that didn't pan out. This looks to be the successor, with fanout replication an explicit feature. Using a FUSE layer that's detecting changes is sure to have some performance tradeoffs. What benchmarks are under way? Do you need any help?

Thanks! In hindsight, I'm glad I pulled the live replication in Litestream and left it focused on disaster recovery only. A lot of what I'm doing in LiteFS would be much more difficult if I didn't have control over the write path. I'm able to perform rolling checksums to verify database integrity across replicas and that would be nearly impossible with Litestream (which runs as a separate process). The FUSE layer doe…

I’m curious, was there a specific reason that FUSE was used instead of the FVS layer inside to SQLite itself? I realize that it would mean that a custom module would have to be loaded, and maybe that was undesirable. Full disclosure I have not read the link yet as I am a little pressed for time, but I will later tonight when I have time. If it’s mentioned in the readme, feel free to ignore me :) I ask because I have been building a VFS for SQLite to handle multiple distributed writers and replicating data blocks around to different places.

Re: LiteFS a FUSE-based file system for replicating SQLite

#34

Post by liuliu »

benbjohnson wrote: post #23
liuliu wrote: post #17

Earlier quoted context omitted.

Does LiteFS needs to do "pattern matching" to know what SQLite writes (a txn or not e.g.)? With Litestream, it seems simply use SQLite API. Just curious how do you think through this.

Transactions start with the creation of the journal file and end when it's unlinked so there's some "pattern matching" but it's not terribly complicated. Once the file is unlinked, LiteFS can read and verify the journal and pull the change set of pages from the database file. Litestream interacts with the SQLite API for locking but it uses a regular file descriptor to read the WAL. It has to do similar parsing of pag…

I am under the impression that if you are in the WAL mode, there is no rollback journal created?

Re: LiteFS a FUSE-based file system for replicating SQLite

#35

Post by benbjohnson »

bbertelsen wrote: post #33
benbjohnson wrote: post #2

LiteFS author here (also Litestream author). I'm happy to answer any questions folks have about how it works or what's on the roadmap.

Can you describe one or more use cases?

One main advantage of LiteFS is being able to deploy a SQLite-based application and not have downtime during deploys. This is especially true in ephemeral systems like Kubernetes where pods will rollover.

Another use case is moving data close to users. If you're only targeting users in a single country then it's not as big of a deal but RTT from the US to Europe is ~100ms and US to Asia is ~250ms. That's a big latency hit depending on what you're trying to do.

Re: LiteFS a FUSE-based file system for replicating SQLite

#36

Post by benbjohnson »

dan353hehe wrote: post #32
benbjohnson wrote: post #8

Earlier quoted context omitted.

Thanks! In hindsight, I'm glad I pulled the live replication in Litestream and left it focused on disaster recovery only. A lot of what I'm doing in LiteFS would be much more difficult if I didn't have control over the write path. I'm able to perform rolling checksums to verify database integrity across replicas and that would be nearly impossible with Litestream (which runs as a separate process). The FUSE layer doe…

I’m curious, was there a specific reason that FUSE was used instead of the FVS layer inside to SQLite itself? I realize that it would mean that a custom module would have to be loaded, and maybe that was undesirable. Full disclosure I have not read the link yet as I am a little pressed for time, but I will later tonight when I have time. If it’s mentioned in the readme, feel free to ignore me :) I ask because I have…

It's a good question. It's mainly about reducing friction for the end user. I wrote up a related response here: https://news.ycombinator.com/item?id=32243305

Re: LiteFS a FUSE-based file system for replicating SQLite

#37

Post by qbasic_forever »

bbertelsen wrote: post #33
benbjohnson wrote: post #2

LiteFS author here (also Litestream author). I'm happy to answer any questions folks have about how it works or what's on the roadmap.

Can you describe one or more use cases?

It might be interesting for embedded scenarios where you need an always-available database but can't guarantee there will be a connection available consistently to a central database server.

Re: LiteFS a FUSE-based file system for replicating SQLite

#38

Post by laweijfmvo »

benbjohnson wrote: post #2

LiteFS author here (also Litestream author). I'm happy to answer any questions folks have about how it works or what's on the roadmap.

A while ago, when I discovered litestream, I played around with it and just setup cloning to a local file://; is this the same idea?

Re: LiteFS a FUSE-based file system for replicating SQLite

#39

Post by ignoramous »

benbjohnson wrote: post #27
ctur wrote: post #21

If a "FUSE to replicate SQLite" solution came from anywhere else, I'd be quite skeptical, but there is a lot of very interesting tech coming out of fly.io these days and Ben certainly knows this space well. It still feels a little like a hack and piercing of layers of abstraction (less so than, say, litestream). I love it when at first glance it isn't clear if a project is a crazy idea from someone just goofing aroun…

Thanks for the vote of confidence! I can understand the "hack" feel -- it's a trade-off. If I wrote it the "proper" way and integrated directly into the SQLite source or used a VFS then it'd be a lot harder to deploy for most folks. By making it a FUSE file system, someone can use it without really knowing much about it from the application's perspective. As for strategy, it unfortunately doesn't work to layer with L…

> I'm adding S3 replication support directly into LiteFS.

Nice! There's a lot of value one can get out of a blob store, despite it appearing seemingly at odds with block-device dependent systems, like most sql dbms.

When a database at BigCloud layered replication (real-time backups) atop S3, they did so by shipping both the WAL and the on-disk files. For write heavy tables, WAL was streamed every second, and on-disk files (snapshots) every 30mins (or at some apt size-based threshold).

While WAL streaming also doubled-up as a key foundation for them to build materialized views, support real-time triggers, and act as an online data-verification layer; S3 itself served as an insurance against hardware errors (memory, cpu, network, disk) and data corruption.

https://web.archive.org/web/20220712155558/https://www.useni... (keyword search S3)

Elasticsearch / OpenSearch does something similar but it only implements snapshot-based replication to S3 (periodic backups).

https://web.archive.org/web/20190722153122/https://www.micro... / https://archive.is/Q5jUj (docs)

Re: LiteFS a FUSE-based file system for replicating SQLite

#40

Post by mandeepj »

benbjohnson wrote: post #20
mandeepj wrote: post #11

Db sharding and replication is a fascinating subject and a matter of deep interest to me. Ben and Matt - appreciate your contributions in this area. I'm interested in making contributions along with you. Please let me know if you are looking for help. Much Thanks.

Thanks! I think sharding is really interesting -- especially with a lightweight database like SQLite. I'm not looking for contributions right now but I would love to hear any feedback on the approach taken with LiteFS. I want to make it as easy to run as possible.

I'd give LiteFS a run soon. SqlLite is cross-platform, but seems like LiteFS is not. True?

https://github.com/superfly/litefs

Post reply on HN