Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LiteFS a FUSE-based file system for replicating SQLite

github.com

Post reply on HN
Print view View original

61–66 of 66 posts

Re: LiteFS a FUSE-based file system for replicating SQLite

#61

Post by EdSchouten »

anon291 wrote: post #16
benbjohnson wrote: post #2

LiteFS author here (also Litestream author). I'm happy to answer any questions folks have about how it works or what's on the roadmap.

Why a fuse filesystem instead of just using sqlite's vfs features? FUSE adds an extra user-kernel-user context switch which is expensive compared to a quick user-user call for the vfs drivers.

I was going to raise that point exactly.

As someone who spends an awful amount of time using FUSE, my recommendation is to only use it in cases where the software that interacts with the file system isn't easily changeable. For example, for Buildbarn which I maintain (https://github.com/buildbarn/bb-remote-execution), I need to use it. It's infeasible to change arbitrary compilers and tests to all interact with a network distributed build cache. Designing the FUSE file system was a pretty heavy investment though, as you really need to be POSIXly correct to make it all work. The quality of implementations of FUSE also varies between OSes and their versions. macFUSE, for example, is quite different from Linux FUSE.

Given that SQLite already has all of the hooks in place, I would strongly recommend using those. In addition to increasing portability, it also makes it easier to set up/run. As an example, it's pretty hard to mount a FUSE file system inside of a container running on Kubernetes without risking locking up the underlying host. Doing the same thing with the SQLite VFS hooks is likely easy and also doesn't require your container to run with superuser privileges.

Re: LiteFS a FUSE-based file system for replicating SQLite

#63

Post by spiffytech »

westurner wrote: post #58
gigatexal wrote: post #57

Earlier quoted context omitted.

SQLite v Postgres here is apples v oranges. Postgres is multi-reader multi-writer whereas SQLite is single-writer multi-reader among other things. They are both very fine databases but solve different use cases.

From https://www.sqlite.org/isolation.html : > Isolation And Concurrency: SQLite implements isolation and concurrency control (and atomicity) using transient journal files that appear in the same directory as the database file. There are two major "journal modes". The older "rollback mode" corresponds to using the "DELETE", "PERSIST", or "TRUNCATE" options to the journal_mode pragma. In rollback mode, changes are wri…

This is saying that SQLite allows reads and writes to happen simultaneously, but it's still single-writer. There's a WIP branch to add concurrent writes.

> Usually, SQLite allows at most one writer to proceed concurrently. The BEGIN CONCURRENT enhancement allows multiple writers to process write transactions simultanously if the database is in "wal" or "wal2" mode, although the system still serializes COMMIT commands.

https://www.sqlite.org/cgi/src/doc/begin-concurrent/doc/begi...

Re: LiteFS a FUSE-based file system for replicating SQLite

#64

Post by ThinkBeat »

mrkurt wrote: post #53
ThinkBeat wrote: post #52

Having been a happy user developing solutions around SQLite for a good amount of time, I find all these "enterprisy" hacks / extension curious. There are great solutions out there that handle these things and have for a long time. I know SQLite has become the new hotness, but I really do not want SQLite to get good at all these things because then it would no longer be great at what it does marvelously already.

This is about as non-enterprisey as it gets. It's built to make sqlite work better for tiny little node.js apps running on very cheap hosting.

From the first of the description of the project on GitHub.

> """LiteFS is a FUSE-based file system for replicating SQLite databases across a cluster of machines."""

> """Leader election: currently implemented by Consul using sessions """

That sounds enterprisy to me.

Re: LiteFS a FUSE-based file system for replicating SQLite

#65

Post by benbjohnson »

ThinkBeat wrote: post #64
mrkurt wrote: post #53

Earlier quoted context omitted.

This is about as non-enterprisey as it gets. It's built to make sqlite work better for tiny little node.js apps running on very cheap hosting.

From the first of the description of the project on GitHub. > """LiteFS is a FUSE-based file system for replicating SQLite databases across a cluster of machines.""" > """Leader election: currently implemented by Consul using sessions """ That sounds enterprisy to me.

I can see Consul feeling enterprise-y. If you want to distribute SQLite without downtime then you need to move the primary around and that requires leader election. We have Consul already running at Fly.io but I feel like a lot of folks have either Consul or etcd running for some part of their system out there so the idea was to be able to attach onto that.

Re: LiteFS a FUSE-based file system for replicating SQLite

#66

Post by vaxman »

LiteFS, a tool for edge-case data loss, unintended service hangs and data corruption /s

this kind of scheme was explored and naturally selected away in the pre-Internet networking era (whose applied knowledge was mostly lost and unavailable to newer generations after the boomer purge that began in the 2000 market crash) .. this kind of scheme should always be isolated to tightly coupled machines operating over fault tolerant and RELATIVELY high-speed links (like a cluster of boards interconnected via PCIe, fibre or Thunderbolt, each with [controller] ECC memory) not between WAN zones.

But in a tightly-coupled cluster-type environment, better solutions would exist like Redis Cluster (and RediSQL) that could be further upgraded with some kind of shared pagefile.

But for testing near-obsolete non-serverless cloud code against edge-cases, consider building a testbed of 4 or 5 rPis talking to each other over (really) slow LoRA links ..then amp up the RFi, thermal and vibration procedures and add all the code to maintain the data integrity as you monitor for the various failure scenarios.

Post reply on HN