Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LiteFS

fly.io

Post reply on HN
Print view View original

81–90 of 158 posts

Re: LiteFS

#81

Post by simonw »

omnimus wrote: post #78
benbjohnson wrote: post #38

Earlier quoted context omitted.

> Where is the data actually being stored in this setup? A copy on each machine running the application? Yes, each node has a full copy of the database locally. > If so, is there another copy somewhere else (e.g. S3) in case all nodes go down? S3 replication support is coming[1]. Probably in the next month or so. Until then, it's recommended that you run a persistent volume with your nodes. > What happens if the Cons…

Will this replicate the filesystem/volume between instances? Can for example use text files or different kind of filebased database? Or it requires me to use sqlite?

This is only for SQLite. But you can put text files (or even binary files - see https://simonwillison.net/2020/Jul/30/fun-binary-data-and-sq...) in a SQLite database and you may find you get better performance than reading files from disk: https://www.sqlite.org/fasterthanfs.html

Re: LiteFS

#82

Post by simonw »

hinkley wrote: post #76
infogulch wrote: post #47

> To improve latency, we're aiming at a scale-out model that works similarly to Fly Postgres. That's to say: writes get forwarded to the primary and all read requests get served from their local copies. How can you ensure that a client that just performed a forwarded write will be able to read that back on their local replica on subsequent reads?

That's a fun one. A couple years ago someone posted a solution to that here. I'm not sure if it works for SQLite, but it worked for Postgres. The basics of it were that each replica was aware of the latest transaction ID it had seen. On a normal read you'd deal with the usual set of eventually consistent issues. But on a read-after-write, you would select a replica that was ahead of the write transaction. Ultimately…

Another solution that I've used successfully in the past for web apps is to set a 10s cookie every time a user performs a write, and then route their read requests to the lead server until that cookie expires. That way they're sure to see the impact of the write they just made.

Re: LiteFS

#83

Post by endisneigh »

Seems neat, until you try to do schema migrations. Unless they can guarantee that all containers’ SQLite instances have the same scheme without locking I’m not sure how doesn’t run into the same issues as many NoSQL.

CouchDB had this same issue with its database per user model and eventually consistent writes.

Re: LiteFS

#84

Post by zie »

tshaddox wrote: post #41
tptacek wrote: post #18

Earlier quoted context omitted.

Anyone want to chime in with the largest app they've deployed where prod Postgres was reachable over a Unix domain socket?

Are there significant limits to the size of an app that could be deployed alongside Postgres versus the size of an app that could be deployed alongside SQLite?

No, the big difference is write speed. SQLite is limited in the amount of writes it can do at the same time(typically 1). This is generally called "concurrent writes". Last I checked SQLite can't really get past 1 write at the same time, but it can emulate concurrent writes in WAL mode such that it isn't normally a problem, for most applications. Postgres has no such limit(though there can be limits to updating a particular row/column of data concurrently, obviously).

Otherwise both are generally limited to the physical resources of the machine(memory, disk, etc). Generally speaking you can scale boxes much much farther than your data size for most applications.

Re: LiteFS

#85

Post by cbsmith »

simonw wrote: post #79
cbsmith wrote: post #70

Earlier quoted context omitted.

Isn't this all talking to S3 anyway, not to mention the network trips intrinsic to the system before it gets to S3? I mean, I'm sure there's some performance win here, but I'm surprised it's so significant. It's not like the address-space separation is without benefits... heck, if it weren't, you could simply have embedded the whole application inside Postgres and achieved the same effect.

You may be confusing Litestream and LiteFS. Litestream writes everything to S3 (or similar storage). LiteFS lets different nodes copy replicated data directly to each other over a network, without involving S3. In either case, the actual SQLite writes and reads all happen directly against local disk, without any network traffic. Replication happens after that.

I think I was definitely confusing it with Litestream as the blog post made reference to it (and I did find that confusing).

That said, unless I've misunderstood the LifeFS use case, you're still going over the network to reach a node, and that node is still going through a FUSE filesystem. That would seem to create overhead comparable (potentially more significant) to talking to a Postgres database hosted on a remote node.

It just doesn't seem that obvious that there's a big performance win here. I'd be curious to see the profiling data behind this.

Re: LiteFS

#86

Post by slively »

endisneigh wrote: post #83

Seems neat, until you try to do schema migrations. Unless they can guarantee that all containers’ SQLite instances have the same scheme without locking I’m not sure how doesn’t run into the same issues as many NoSQL. CouchDB had this same issue with its database per user model and eventually consistent writes.

In my experience distributed database migrations tend to be two phase though so it can be compatible during the migration. The first migration to add something, release the apps to use it, then a second migration to cleanup.

Re: LiteFS

#87

Post by toast0 »

theomega wrote: post #58
toast0 wrote: post #27

Earlier quoted context omitted.

Could easily be a path MTU issue? I don't currently have working IPv6, but on TCP/IPv4, the server->client handshake packets I get back from fly.io are full length packets. If your tcpv6 MSS is set incorrectly, and some other not great things are happening in the path, then you might not be able to receive large packets.

How could I find out if this is the case? And, what can I do about it? Disabling IPv6 obviously fixes it, but that it not a solution...

I'm not aware of a simple to use test site for this on IPv6 (I've got one for IPv4, but my server host doesn't do IPv6 either, and using a tunnel will limit the ranges I can test, etc)... so you'll need to test ye olde fashioned way.

I don't have a mac, but if the man page[1] is right, something like this should work to see how big of a packet you can successfully send and receive:

      ping -6 -D -G 1500 -g 1400 fly.io
(you may need to run as root to set packet sizes). You should get back a list of replies with say 1408 bytes, then 1409, etc. The last number you get back is effectively the largest IPv6 payload you can receive (on this path, it could be different for other paths), and if you add the IPv6 header length of 40, that's your effective path MTU.

Use tcpdump to see what TCP MSS is being sent on your outgoing SYN packets, for IPv4, the MSS is MTU - 40 (20 for IPV4 header, 20 for TCP header), for IPv6, the MSS should be MTU - 60 (40 for IPv6 header, 20 for TCP header). If your TCP MSS is higher than the observed path MTU to fly.io, that's likely the immediate cause of your problem.

If you're using a router, make sure it knows the proper MTU for the IPv6 connection, and enable MSS clamping on IPv6, if possible --- or make sure the router advertisement daemon shares the correct MTU.

Hope this gets you started.

[1] https://ss64.com/osx/ping.html

Re: LiteFS

#88

Post by hinkley »

simonw wrote: post #82
hinkley wrote: post #76

Earlier quoted context omitted.

That's a fun one. A couple years ago someone posted a solution to that here. I'm not sure if it works for SQLite, but it worked for Postgres. The basics of it were that each replica was aware of the latest transaction ID it had seen. On a normal read you'd deal with the usual set of eventually consistent issues. But on a read-after-write, you would select a replica that was ahead of the write transaction. Ultimately…

Another solution that I've used successfully in the past for web apps is to set a 10s cookie every time a user performs a write, and then route their read requests to the lead server until that cookie expires. That way they're sure to see the impact of the write they just made.

If write traffic is low enough I could see that working, but 10s is an awful long time. Anything less than a 1000:1 read:write ratio would start running into capacity problems wouldn't it?

Re: LiteFS

#89

Post by threatofrain »

benbjohnson wrote: post #5
no_wizard wrote: post #2

This is distributed SQLite 3, running (I assume at least partially managed?) LiteFS[5] for you. Which is pretty cool! What I'd like to have seen is how this compares to things like rqlite[1] or Cloudflare's D1[2] addressed directly in the article That said, I think this is pretty good for things like read replica's. I know the sales pitch here is as a full database, and I don't disagree with it, and if I was starting…

LiteFS/Litestream author here. You bring up a lot of great points that I'll try to address. > What I'd like to have seen is how this compares to things like rqlite or Cloudflare's D1 addressed directly in the article I think a post comparing the different options is a great idea. I'll try to summarize a bit here though. LiteFS aims to be an analogue to Postgres replication but with built-in failover. Postgres uses lo…

Cloudflare D1 is in closed beta so everything is presumably tentative, but atm they don't support transactions (!) and it doesn't appear to be on their near-term product roadmap. That one really surprised me.

Re: LiteFS

#90

Post by miduil »

seabrookmx wrote: post #71
miduil wrote: post #52

Earlier quoted context omitted.

I think FUSE performance has significantly improved since 2010, the year in which ZFS-On-Linux became available I believe. This is just one of many examples: https://www.phoronix.com/news/MTI1MzM In contrary, in some cases FUSE is even faster than doing a regular kernel mount(). There is an experimental research distribution called distri that is exclusively relying on fuse mounting and figured out that FUSE was fast…

You're conflating zfs-fuse and zfs-on-linux (ZoL). The first one is as it's name suggests.. a user space implementation of ZFS. ZoL (now unified with OpenZFS) is implemented as a kernel module and as such does _not_ run in user space. It performs significantly better as a result. FUSE is still slow, which is why there's ongoing effort to replace things like NTFS-3G (the default NTFS implementation in most linux distr…

How do I conflate zfs-fuse with ZoL? I was trying to highlight that if someones memory of "poor fuse performance" is because of zfs-fuse, then someones memory must be quite old, as ZoL has been available for quite some time and since then also fuse itself has received quite some performance gains.

Edit: Also I don't want to imply that FUSE is near in-kernel filesystems, but it is certainly performing much better than 12 years ago.

Post reply on HN