Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LiteFS

fly.io

Post reply on HN
Print view View original

101–110 of 158 posts

Re: LiteFS

#101

Post by benbjohnson »

simonw wrote: post #69
vcryan wrote: post #59

This approach is very appealing to me :) curious about how people handle schema migrations when using this approach. I segment sqlite files (databases) that have the same schema into the same folder. I haven't really had a case where migrations was really a concern, but I could see it happening soon. Seems like in my deployment, I'm going to need an approach to loop over dbs to apply this change... I currently have a…

Yeah schema migrations are going to be interesting. Presumably those end up making a HUGE change to the WAL (since they could affect every stored row in a large table), which means LiteFS then has to send a truly giant chunk of data out to the replicas. I wonder how this would present itself in real-world usage? Would the replicas go out-of-date for 10-30s but continue serving read-only traffic, or could there be som…

> Presumably those end up making a HUGE change to the WAL (since they could affect every stored row in a large table), which means LiteFS then has to send a truly giant chunk of data out to the replicas.

Yes, a large migration could end up sending a lot of data out to replicas. One way to mitigate this is to shard your data into separate SQLite databases so you're only migrating a small subset at a time. Of course, that doesn't work for every application.

> Would the replicas go out-of-date for 10-30s but continue serving read-only traffic, or could there be some element of cluster downtime caused by this?

Once WAL support is in LiteFS, it will be able to replicate this out without causing any read down time. SQLite is still a single-writer database though so writes would be blocked during a migration.

Re: LiteFS

#102

Post by benbjohnson »

hinkley wrote: post #76
infogulch wrote: post #47

> To improve latency, we're aiming at a scale-out model that works similarly to Fly Postgres. That's to say: writes get forwarded to the primary and all read requests get served from their local copies. How can you ensure that a client that just performed a forwarded write will be able to read that back on their local replica on subsequent reads?

That's a fun one. A couple years ago someone posted a solution to that here. I'm not sure if it works for SQLite, but it worked for Postgres. The basics of it were that each replica was aware of the latest transaction ID it had seen. On a normal read you'd deal with the usual set of eventually consistent issues. But on a read-after-write, you would select a replica that was ahead of the write transaction. Ultimately…

LiteFS provides a transaction ID that applications can use to determine replication lag. If the replica is behind the TXID, it can either wait or it can forward to the primary to ensure consistency.

Re: LiteFS

#103

Post by benbjohnson »

sk55 wrote: post #75
infogulch wrote: post #47

> To improve latency, we're aiming at a scale-out model that works similarly to Fly Postgres. That's to say: writes get forwarded to the primary and all read requests get served from their local copies. How can you ensure that a client that just performed a forwarded write will be able to read that back on their local replica on subsequent reads?

They don't have ACID guarantees with this setup.

ACID is typically used to describe single database instances and from that perspective, LiteFS doesn't change the ACID semantics that SQLite has. For distributed consistency models[1], it gets more complicated as LiteFS currently just supports async replication. We'll introduce stronger guarantees in the future although we currently provide some information such as transaction IDs so you can manage replication lag if you need stronger consistency.

[1]: https://jepsen.io/consistency

Re: LiteFS

#104

Post by benbjohnson »

jensneuse wrote: post #74

Sound like a drop in solution to add high availability to WunderBase ( https://github.com/wundergraph/wunderbase ). Can we combine LiteFS with Litestream for Backups, or how would you do HA + Backups together?

Yes, it should fit with WunderBase to provide HA. I'm adding S3 support which will work the same (in principle) as Litestream. I'm hoping to have that in about a month or so.

Re: LiteFS

#105

Post by cbsmith »

simonw wrote: post #93
cbsmith wrote: post #85

Earlier quoted context omitted.

I think I was definitely confusing it with Litestream as the blog post made reference to it (and I did find that confusing). That said, unless I've misunderstood the LifeFS use case, you're still going over the network to reach a node, and that node is still going through a FUSE filesystem. That would seem to create overhead comparable (potentially more significant) to talking to a Postgres database hosted on a remot…

The performance boost that matters most here is when your application reads from the database. Your application code is reading directly from disk there, through a very thin FUSE layer that does nothing at all with reads (it only monitors writes). So your read queries should mostly be measured in microseconds.

> So your read queries should mostly be measured in microseconds.

You should check out the read latency for read-only requests over unix domain sockets with PostgreSQL. You tend to measure it in microseconds, and depending on circumstances it can be single-digit microseconds.

Regardless of whether your FUSE logic does nothing at all, It sure seems like there's intrinsic overhead to the FUSE model that is very similar to the intrinsic overhead of talking to another userspace database process... because you're talking to another userspace process (through the VFS layer).

When the application reads, those requests go from userspace to the FUSE driver & /dev/fuse, with the thread being put into a wait state; then the FUSE daemon needs to pull the request from /dev/fuse to service it; then your FUSE code does whatever minimal work it needs to do to process the read and passes it back through /dev/fuse and the FUSE driver, and from there back to your application. That gets you pretty much the same "block and context switch" overhead of an IPC call to Postgres (arguably more). FUSE uses splicing to minimize data copying (of course, unix domain sockets also minimize data copying), though looking at the LiteFS Go daemon, I'm not entirely sure there isn't a copy going on anyway. Memory copying issues aside, from a latency perspective, you're jumping through very similar hoops to talking to another user-space process... because that's how FUSE works.

There's a potential performance win if the data you're reading is already in the VFS cache, since that would bypass having to through the FUSE filesystem (and the /dev/fuse-to-userspace jump) entirely. The catch is, at that point you're bypassing SQLite transaction engine semantics entirely, giving you a dirty read that's really just a cached result from a previous read. That's not really a new trick, and you can get even better performance with client-side caching that can avoid a trip to kernel space.

I'm sure there's a win here somewhere, but I'm struggling to understand where.

Re: LiteFS

#106

Post by rsync »

benbjohnson wrote: post #104
jensneuse wrote: post #74

Sound like a drop in solution to add high availability to WunderBase ( https://github.com/wundergraph/wunderbase ). Can we combine LiteFS with Litestream for Backups, or how would you do HA + Backups together?

Yes, it should fit with WunderBase to provide HA. I'm adding S3 support which will work the same (in principle) as Litestream. I'm hoping to have that in about a month or so.

Can you comment on how you added SFTP support to litestream so that it would work with rsync.net[1][2] ?

How does that compare/contrast with what my grandparent is alluding to ?

[1] https://github.com/benbjohnson/litestream/issues/140

[2] https://www.rsync.net/resources/notes/2021-q3-rsync.net_tech...

Re: LiteFS

#107

Post by boltzmann-brain »

benbjohnson wrote: post #19
ranger_danger wrote: post #13

ELI5?

LiteFS makes it so that you can have a SQLite database that is transparently replicated to a cluster of machines. The use case is if you have your app running on a server running in Chicago, then users in Europe will have 100ms latency to your server and users in Asia will have a 250ms latency. That's on top of the time it takes for your server to process the request. Many people target sub-100ms latency to make thei…

> Traditionally, it's complicated to replicate your database to different regions of the world using something like Postgres

what makes it complicated?

Re: LiteFS

#108

Post by cbsmith »

WorldMaker wrote: post #97
cbsmith wrote: post #85

Earlier quoted context omitted.

I think I was definitely confusing it with Litestream as the blog post made reference to it (and I did find that confusing). That said, unless I've misunderstood the LifeFS use case, you're still going over the network to reach a node, and that node is still going through a FUSE filesystem. That would seem to create overhead comparable (potentially more significant) to talking to a Postgres database hosted on a remot…

It's an "eventual consistency" transaction scheme for SQLite: reads happen locally and immediately with the nearest edge copy available, it's only writes that need to be forwarded to a (potentially far away) primary DB over the network somewhere, and transactions will then take some amount of network time to fan back out to edges, so eventually the edges will read the latest transactions, but during that fan out will…

Local for the application code on the node you're talking to (just like the read would be local for a Postgres process running on the node), but there's still a network trip to get to the node, no?

Even if the reads are happening locally, if they're going through FUSE (even a "thin" pass-through that does nothing), that means they're getting routed from kernel space to a FUSE daemon, which means you're still doing IPC to another process through a kernel...

Re: LiteFS

#109

Post by BeefWellington »

tptacek wrote: post #18
pphysch wrote: post #16

> Developing against a relational database requires devs to watch out for "N+1" query patterns, where a query leads to a loop that leads to more queries. N+1 queries against Postgres and MySQL can be lethal to performance. Not so much for SQLite. This is misleading AFAICT. The article(s) is actually comparing remote RDBMS to local RDBMS, not Postgres to SQLite. Postgres can also be served over a UNIX socket, removing…

Anyone want to chime in with the largest app they've deployed where prod Postgres was reachable over a Unix domain socket?

Back in my consulting days one of my clients had an application with nearly all of the business logic being done in-database. The app was essentially a frontend for stored procedures galore and it sat on an absolute monster of a system for its time (~256GB of RAM, 24 or 32 Cores I think). It handled something like 100k DAU with no issue. That was for just the transactional system. They replicated data to other (similar style) systems for less mission-critical things like reporting, billing, etc.

I want to be clear though, I would never recommend anyone do this. For its time it was impressive but I suspect by now it's been re-engineered away from that model, if for no other reason than its a lot of eggs in one basket.

Re: LiteFS

#110

Post by simonw »

cbsmith wrote: post #105
simonw wrote: post #93

Earlier quoted context omitted.

The performance boost that matters most here is when your application reads from the database. Your application code is reading directly from disk there, through a very thin FUSE layer that does nothing at all with reads (it only monitors writes). So your read queries should mostly be measured in microseconds.

> So your read queries should mostly be measured in microseconds. You should check out the read latency for read-only requests over unix domain sockets with PostgreSQL. You tend to measure it in microseconds, and depending on circumstances it can be single-digit microseconds. Regardless of whether your FUSE logic does nothing at all, It sure seems like there's intrinsic overhead to the FUSE model that is very similar…

That's a really detailed, interesting reply. I am now really interested to understand more about the comparative overhead here.
Post reply on HN