Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LiteFS

fly.io

Post reply on HN
Print view View original

121–130 of 158 posts

Re: LiteFS

#121

Post by benbjohnson »

cbsmith wrote: post #105
simonw wrote: post #93

Earlier quoted context omitted.

The performance boost that matters most here is when your application reads from the database. Your application code is reading directly from disk there, through a very thin FUSE layer that does nothing at all with reads (it only monitors writes). So your read queries should mostly be measured in microseconds.

> So your read queries should mostly be measured in microseconds. You should check out the read latency for read-only requests over unix domain sockets with PostgreSQL. You tend to measure it in microseconds, and depending on circumstances it can be single-digit microseconds. Regardless of whether your FUSE logic does nothing at all, It sure seems like there's intrinsic overhead to the FUSE model that is very similar…

> There's a potential performance win if the data you're reading is already in the VFS cache, since that would bypass having to through the FUSE filesystem.

That's a typical case for a lot of databases. Most of the "hot" data is in a small subset of the pages and many of those can live in the in-process page cache.

> The catch is, at that point you're bypassing SQLite transaction engine semantics entirely, giving you a dirty read that's really just a cached result from a previous read.

It's not bypassing the transaction engine semantics. For WAL mode, SQLite can determine when pages are updated by checking the SHM file and then reading updated pages from the WAL file. Pages in the cache don't need to flushed on every access or even between transactions to be valid.

> I'm sure there's a win here somewhere, but I'm struggling to understand where.

The main goal of LiteFS is to make it easy to globally replicate applications. Many apps run in a single region of the US (e.g. us-east-1) and that's fast for Americans but it's a 100ms round trip to Europe and a 250ms round trip to Asia. Sure, you can spin up a multi-region Postgres but it's not that easy and you'll likely deploy as separate database and application servers because Postgres is not very lightweight.

LiteFS aims to have a minimal footprint so it makes it possible to deploy many small instances since SQLite is built to run on low resource hardware.

As far as comparisons with Postgres over UNIX sockets, I agree that the performance of a single instance is probably comparable with a FUSE layer.

Re: LiteFS

#122

Post by benbjohnson »

Spivak wrote: post #120
benbjohnson wrote: post #99

Earlier quoted context omitted.

Good question. Right now, LiteFS operates with async replication so its possible to have a transaction written to the primary get lost if the primary fails before it's replicated out. For that short window, you could read a transaction that no longer exists. From that standpoint, I believe it would technically be Read Uncommitted. However, during normal operation it'll function more like Snapshot Isolation. LiteFS do…

I think you’re being too harsh on yourself, isolation levels don’t typically account for replication lag and network partitioning into account. For example on MySQL you can have async replication and serializable isolation. Replicas in this mode might never receive updates.

Isolation levels in ACID are typically used for single instances. However, if you're talking about a distributed system then consistency diagram on Jepsen is more applicable[1]. Raft, for example, can ensure read consistency across a cluster by running reads through the consensus mechanism.

LiteFS aims to be the analogue to Postgres/MySQL replication and those generally work great for most applications.

[1]: https://jepsen.io/consistency

Re: LiteFS

#123

Post by tptacek »

ignoramous wrote: post #45
asim wrote: post #7

10 years ago fly.io is the company I wanted to build. Something with massive technical depth that becomes a developer product. They're doing an incredible job and part of that comes down to how they evangelise the product outside of all the technical hackery. This requires so much continued effort. AND THEN to actually run a business on top of all that. Kudos to you guys. I struggled so much with this. Wish you nothi…

5 years ago, fly.io was basically poor man's deno.com / oven.sh [0]. In my (incorrect) opinion, tptacek changed fly.io's trajectory single-handedly. [0] https://ghostarchive.org/varchive/r-1hXDvOoHA

No? No. No!

Re: LiteFS

#124

Post by hinkley »

> Second, your application can only serve requests from that one server. If you fired up your server in Dallas then that'll be snappy for Texans. But your users in Chennai will be cursing your sluggish response times since there's a 250ms ping time between Texas & India.

> To improve availability, it uses leases to determine the primary node in your cluster. By default, it uses Hashicorp's Consul.

Having a satellite office become leader of a cluster is one of the classic blunders in distributed computing.

There are variants of Raft where you can have quorum members that won't nominate themselves for election, but out of the box this is a bad plan.

If you have a Dallas, Chennai, Chicago, and Cleveland office and Dallas goes dark (ie, the tunnel gets fucked up for the fifth time this year), you want Chicago to become the leader, Cleveland if you're desperate. But if Chennai gets elected then everyone has a bad time, including Dallas when it comes back online.

Re: LiteFS

#125

Post by rsync »

benbjohnson wrote: post #116
rsync wrote: post #106

Earlier quoted context omitted.

Can you comment on how you added SFTP support to litestream so that it would work with rsync.net[1][2] ? How does that compare/contrast with what my grandparent is alluding to ? [1] https://github.com/benbjohnson/litestream/issues/140 [2] https://www.rsync.net/resources/notes/2021-q3-rsync.net_tech...

Litestream does log shipping so it just takes a chunk of the write-ahead log (WAL) and copies it out a destination. That can be S3, GCP, SFTP, etc. LiteFS will do the same although I'm only targeting S3 initially. That seemed to be the bulk of what people used.

OK, thank you.

As with the litestream component, it would be wonderful to have an SFTP transport for LiteFS.

When you say 'S3' I assume you mean "S3 compatible API" so that would be fairly open and portable but SFTP would be even more so.

Thanks again.

Re: LiteFS

#126

Post by Scarbutt »

benbjohnson wrote: post #31
pphysch wrote: post #16

> Developing against a relational database requires devs to watch out for "N+1" query patterns, where a query leads to a loop that leads to more queries. N+1 queries against Postgres and MySQL can be lethal to performance. Not so much for SQLite. This is misleading AFAICT. The article(s) is actually comparing remote RDBMS to local RDBMS, not Postgres to SQLite. Postgres can also be served over a UNIX socket, removing…

You could run Postgres over UNIX sockets although you will still get higher latency than SQLite's in-process model. Also, running a Postgres on every app instance on the edge probably isn't practical. Postgres has some great advanced features if you need them but it's also much more heavy weight. With LiteFS, we're aiming to easily run on low resource cloud hardware such as nodes with 256MB or less of RAM. I haven't…

TL;DR "Postgres doesn't fit well our edge services business model" which sure fine but the article is indeed biased/misleading by completely ignoring and not mentioning the option of running postgres and app in a single server.

The gains in latency with sqlite won't matter as soon as throughput starts to dominate.

Re: LiteFS

#127

Post by benbjohnson »

hinkley wrote: post #124

> Second, your application can only serve requests from that one server. If you fired up your server in Dallas then that'll be snappy for Texans. But your users in Chennai will be cursing your sluggish response times since there's a 250ms ping time between Texas & India. > To improve availability, it uses leases to determine the primary node in your cluster. By default, it uses Hashicorp's Consul. Having a satellite…

Good catch. LiteFS has an option to mark nodes as candidates or not. For example, you can run a cluster with a couple candidate nodes in one region (e.g Chicago) and every other region may just run a single non-electable replica.

We don’t support any kind of tiering for candidates in different regions. That’s not a bad idea though.

Re: LiteFS

#128

Post by cbsmith »

benbjohnson wrote: post #121
cbsmith wrote: post #105

Earlier quoted context omitted.

> So your read queries should mostly be measured in microseconds. You should check out the read latency for read-only requests over unix domain sockets with PostgreSQL. You tend to measure it in microseconds, and depending on circumstances it can be single-digit microseconds. Regardless of whether your FUSE logic does nothing at all, It sure seems like there's intrinsic overhead to the FUSE model that is very similar…

> There's a potential performance win if the data you're reading is already in the VFS cache, since that would bypass having to through the FUSE filesystem. That's a typical case for a lot of databases. Most of the "hot" data is in a small subset of the pages and many of those can live in the in-process page cache. > The catch is, at that point you're bypassing SQLite transaction engine semantics entirely, giving you…

> That's a typical case for a lot of databases. Most of the "hot" data is in a small subset of the pages and many of those can live in the in-process page cache.

Yes, though most database engines end up managing their own cache and using direct IO, rather than the VFS cache.

> It's not bypassing the transaction engine semantics. For WAL mode, SQLite can determine when pages are updated by checking the SHM file and then reading updated pages from the WAL file. Pages in the cache don't need to flushed on every access or even between transactions to be valid.

That sounds a lot like at least the SHM & WAL checks wouldn't be cached by the VFS, but as I've been looking at the design more carefully, I'm starting to think I understand the idea here. Basically, the SHM & WAL get updated separately, so you might read a stale version, but since you aren't elected to be a writer, that just means you're looking at stale data, not creating an integrity problem.

> The main goal of LiteFS is to make it easy to globally replicate applications. Many apps run in a single region of the US (e.g. us-east-1) and that's fast for Americans but it's a 100ms round trip to Europe and a 250ms round trip to Asia. Sure, you can spin up a multi-region Postgres but it's not that easy and you'll likely deploy as separate database and application servers because Postgres is not very lightweight.

So, I get that multi-region Postgres can be tricky to set up, if you're doing multi-leader, but this seems about as complicated as a "single leader, many followers" set up, and given that what you're trying to do is shave off the hundreds of milliseconds from partially circumnavigating the earth at the speed of light, I'm not sure the perceived performance differences are significant (or really even measurable) compared to fluctuations in network latency of requests to the region-local node.

> LiteFS aims to have a minimal footprint so it makes it possible to deploy many small instances since SQLite is built to run on low resource hardware.

This part I'm getting and the objective a lot of sense to me (and certainly running local postgres instances on every node wouldn't be an obvious approach to me). I hadn't thought of FUSE + SQLite as a way to get there, so this is an interesting and surprising approach. I'm looking forward to how this plays out.

> As far as comparisons with Postgres over UNIX sockets, I agree that the performance of a single instance is probably comparable with a FUSE layer.

Interesting. I was thinking I was missing something. Thanks for all the insight.

Re: LiteFS

#129

Post by hinkley »

benbjohnson wrote: post #38
nicoburns wrote: post #37

Where is the data actually being stored in this setup? A copy on each machine running the application? If so, is there another copy somewhere else (e.g. S3) in case all nodes go down? Also, what happens if the Consul instance goes down? If my application nodes can't be ephemeral then this seems like it would be harder to operate than Postgres or MySQL in practice. If it completely abstracts that away somehow then I s…

> Where is the data actually being stored in this setup? A copy on each machine running the application? Yes, each node has a full copy of the database locally. > If so, is there another copy somewhere else (e.g. S3) in case all nodes go down? S3 replication support is coming[1]. Probably in the next month or so. Until then, it's recommended that you run a persistent volume with your nodes. > What happens if the Cons…

> If Consul goes down then you'll lose write availability but your nodes will still be able to perform read queries.

That's a tricky choice of words, since it looks like you lose Consistency, while retaining Availability and Partition Tolerance. If Consul is down everyone reads stale data, but no writes. Right?

Of course, it's harder for Consul to go down than it is for your database to go down, so the Venn Diagram of "Consul unhappy, Database Happy" is fairly heavily populated with "user error". Which is why 'when in doubt, use Consul' is not terrible advice.

Re: LiteFS

#130

Post by azlyrics »

I actually gave fly.io a whirl over the weekend. was not fun. spent a lot of time on the forums and its pretty clear it has some way to go before it can give AWS or Linode a run for their money.

For instance, we run kubernetes on multiple VPS providers, including public clouds with serverless onramp/offramps deployed on edge location. Anything under 15 minutes are processed by serverless. Anything longer is offloaded to one of the VPS containers available in every part of the world.

I have some more feedbacks ready if you are interested, its a neat idea but not exactly as seamless and easy as the idea proposed since public clouds already offer a way to do this.

Post reply on HN