Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LiteFS

fly.io

Post reply on HN
Print view View original

91–100 of 158 posts

Re: LiteFS

#91

Post by infogulch »

hinkley wrote: post #76
infogulch wrote: post #47

> To improve latency, we're aiming at a scale-out model that works similarly to Fly Postgres. That's to say: writes get forwarded to the primary and all read requests get served from their local copies. How can you ensure that a client that just performed a forwarded write will be able to read that back on their local replica on subsequent reads?

That's a fun one. A couple years ago someone posted a solution to that here. I'm not sure if it works for SQLite, but it worked for Postgres. The basics of it were that each replica was aware of the latest transaction ID it had seen. On a normal read you'd deal with the usual set of eventually consistent issues. But on a read-after-write, you would select a replica that was ahead of the write transaction. Ultimately…

Yeah a transaction ID solution would work pretty well. Every request would return the highest transaction ID seen by the replica; writes forwarded to the primary would also return the transaction ID after that write; any request with a transaction ID higher than the replica would be forwarded to the primary.

Re: LiteFS

#92

Post by hinkley »

infogulch wrote: post #91
hinkley wrote: post #76

Earlier quoted context omitted.

That's a fun one. A couple years ago someone posted a solution to that here. I'm not sure if it works for SQLite, but it worked for Postgres. The basics of it were that each replica was aware of the latest transaction ID it had seen. On a normal read you'd deal with the usual set of eventually consistent issues. But on a read-after-write, you would select a replica that was ahead of the write transaction. Ultimately…

Yeah a transaction ID solution would work pretty well. Every request would return the highest transaction ID seen by the replica; writes forwarded to the primary would also return the transaction ID after that write; any request with a transaction ID higher than the replica would be forwarded to the primary.

I think that may have been the case. Try the local replica, if it is too old, hit the primary, rather than shopping around for the next best replica.

Re: LiteFS

#93

Post by simonw »

cbsmith wrote: post #85
simonw wrote: post #79

Earlier quoted context omitted.

You may be confusing Litestream and LiteFS. Litestream writes everything to S3 (or similar storage). LiteFS lets different nodes copy replicated data directly to each other over a network, without involving S3. In either case, the actual SQLite writes and reads all happen directly against local disk, without any network traffic. Replication happens after that.

I think I was definitely confusing it with Litestream as the blog post made reference to it (and I did find that confusing). That said, unless I've misunderstood the LifeFS use case, you're still going over the network to reach a node, and that node is still going through a FUSE filesystem. That would seem to create overhead comparable (potentially more significant) to talking to a Postgres database hosted on a remot…

The performance boost that matters most here is when your application reads from the database. Your application code is reading directly from disk there, through a very thin FUSE layer that does nothing at all with reads (it only monitors writes).

So your read queries should mostly be measured in microseconds.

Re: LiteFS

#94

Post by nicoburns »

zie wrote: post #84
tshaddox wrote: post #41

Earlier quoted context omitted.

Are there significant limits to the size of an app that could be deployed alongside Postgres versus the size of an app that could be deployed alongside SQLite?

No, the big difference is write speed. SQLite is limited in the amount of writes it can do at the same time(typically 1). This is generally called "concurrent writes". Last I checked SQLite can't really get past 1 write at the same time, but it can emulate concurrent writes in WAL mode such that it isn't normally a problem, for most applications. Postgres has no such limit(though there can be limits to updating a par…

> it can emulate concurrent writes in WAL mode

Seems a bit unfair to call WAL mode emulation of "true" concurrent writes as I'm pretty sure a write-ahead-log (WAL) is exactly how other databases implement multiple concurrent writes. It's just always-on rather than being opt-in.

Re: LiteFS

#95

Post by theomega »

toast0 wrote: post #87
theomega wrote: post #58

Earlier quoted context omitted.

How could I find out if this is the case? And, what can I do about it? Disabling IPv6 obviously fixes it, but that it not a solution...

I'm not aware of a simple to use test site for this on IPv6 (I've got one for IPv4, but my server host doesn't do IPv6 either, and using a tunnel will limit the ranges I can test, etc)... so you'll need to test ye olde fashioned way. I don't have a mac, but if the man page[1] is right, something like this should work to see how big of a packet you can successfully send and receive: ping -6 -D -G 1500 -g 1400 fly.io (…

Thanks, that was of great help!

Just for reference, the ping command is a little different

  sudo ping6 -D -G 1500,1400 fly.io
I set an MSS to 1492 which pfsense (my router) translates to an MSS clamp of 1492-60 for IPv6 and 1492-40 for IPv4. This is a German Deutsche Telekom Fiber connection. Now everything works fine, I can request fly.io (and also discovered that https://ipv6-test.com was not working before and now does with the MSS clamping)

Does MSS clamping have any disadvantages? Are there any alternatives in my case?

Re: LiteFS

#96

Post by whoisjuan »

dcre wrote: post #32
whoisjuan wrote: post #22

Unrelated, but why does the map on their homepage show a region in Cuba? That must be wrong.

I think it's in Miami and the icon is confusingly centered over it. The icons look like pins but I think they're supposed to be hot-air balloons. This explains why some of them appear to point to spots in the ocean. https://fly.io/docs/reference/regions/

Ahh. Great catch. Their balloon definitely looks like a pin.

Re: LiteFS

#97

Post by WorldMaker »

cbsmith wrote: post #85
simonw wrote: post #79

Earlier quoted context omitted.

You may be confusing Litestream and LiteFS. Litestream writes everything to S3 (or similar storage). LiteFS lets different nodes copy replicated data directly to each other over a network, without involving S3. In either case, the actual SQLite writes and reads all happen directly against local disk, without any network traffic. Replication happens after that.

I think I was definitely confusing it with Litestream as the blog post made reference to it (and I did find that confusing). That said, unless I've misunderstood the LifeFS use case, you're still going over the network to reach a node, and that node is still going through a FUSE filesystem. That would seem to create overhead comparable (potentially more significant) to talking to a Postgres database hosted on a remot…

It's an "eventual consistency" transaction scheme for SQLite: reads happen locally and immediately with the nearest edge copy available, it's only writes that need to be forwarded to a (potentially far away) primary DB over the network somewhere, and transactions will then take some amount of network time to fan back out to edges, so eventually the edges will read the latest transactions, but during that fan out will read the previous transaction state.

Re: LiteFS

#98

Post by toast0 »

theomega wrote: post #95
toast0 wrote: post #87

Earlier quoted context omitted.

I'm not aware of a simple to use test site for this on IPv6 (I've got one for IPv4, but my server host doesn't do IPv6 either, and using a tunnel will limit the ranges I can test, etc)... so you'll need to test ye olde fashioned way. I don't have a mac, but if the man page[1] is right, something like this should work to see how big of a packet you can successfully send and receive: ping -6 -D -G 1500 -g 1400 fly.io (…

Thanks, that was of great help! Just for reference, the ping command is a little different sudo ping6 -D -G 1500,1400 fly.io I set an MSS to 1492 which pfsense (my router) translates to an MSS clamp of 1492-60 for IPv6 and 1492-40 for IPv4. This is a German Deutsche Telekom Fiber connection. Now everything works fine, I can request fly.io (and also discovered that https://ipv6-test.com was not working before and now…

Excellent, happy to help, glad you figured out the right command!. 1492 MTU is consistent with PPPoE, or a network where they didn't want to run a separate MTU for PPPoE and straight ethernet.

The only downside to MSS clamping is the computational expense of inspecting and modifying the packets. On a residential connection, where you're running pfsense already, it's probably not even noticeable; but your ISP wouldn't be able to do clamping for you, because large scale routers don't have the processing budget to inspect packets at that level. I've seen some MSS clamping implementations that only clamp packets going out to the internet, and not the return packets... that can lead to problems sending large packets (which isn't always very noticeable, actually; a lot of basic browsing doesn't send packets large enough to hit this, unless you go to a site that sets huge cookies or do some real uploading)

The alternative would be to run a 1492 MTU on your LAN, but that has the marginal negative of reducing your maximum packet size for LAN to LAN transfers.

Re: LiteFS

#99

Post by benbjohnson »

ignoramous wrote: post #63
benbjohnson wrote: post #5

Earlier quoted context omitted.

LiteFS/Litestream author here. You bring up a lot of great points that I'll try to address. > What I'd like to have seen is how this compares to things like rqlite or Cloudflare's D1 addressed directly in the article I think a post comparing the different options is a great idea. I'll try to summarize a bit here though. LiteFS aims to be an analogue to Postgres replication but with built-in failover. Postgres uses lo…

Thanks. > LiteFS still maintains serializable isolation within a transaction, although, it has looser guarantees across nodes than something like rqlite. Picking up a term from the consistency map here [0], what guarantees LiteFS makes across nodes? [0] https://jepsen.io/consistency

Good question. Right now, LiteFS operates with async replication so its possible to have a transaction written to the primary get lost if the primary fails before it's replicated out. For that short window, you could read a transaction that no longer exists. From that standpoint, I believe it would technically be Read Uncommitted.

However, during normal operation it'll function more like Snapshot Isolation. LiteFS does provide a transaction ID so requests could wait for a replica to catch up before issuing a transaction to get something closer to Serializable.

Re: LiteFS

#100

Post by nicoburns »

dspillett wrote: post #40
woooooo wrote: post #23

Earlier quoted context omitted.

Isn't "n + 1" typically a degenerate case of ORM rather than a relational thing? The whole point of rdbms is you can do a join instead of iterated lookups.

You see it in direct code too, from new or naive coders (the good old "it was fast enough on my test data…") or when proof-of-concept code "accidentally" escapes into production. Even within the database you'll sometimes find examples: stored procedures looping over a cursor and running queries or other statements in each iteration. This doesn't have the same latency implications as N+1 requests over a network, but s…

> You see it in direct code too, from new or naive coders

Many moons ago in one of my first coding projects I not only used a separate query for each record, but a separate database connection too. In addition to this, I didn't use a loop to generate these queries, I copy and pasted the connection setup and query code for each record, leading to a 22k code file just for this single web page.

There were probably on the order of 1000 records (they represented premier league football players), and amazingly this code was fast enough (took a couple of seconds to load - which wasn't so unusual back in 2007) and worked well enough that it actually had ~100 real users (not bad a 14 year old's side project).

Post reply on HN