Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LiteFS

fly.io

Post reply on HN
Print view View original

71–80 of 158 posts

Re: LiteFS

#71

Post by seabrookmx »

miduil wrote: post #52
mwcampbell wrote: post #48

I wonder if using FUSE has had any appreciable impact on performance, particularly read performance. I ask because FUSE has historically had a reputation for being slow, e.g. with the old FUSE port of ZFS.

I think FUSE performance has significantly improved since 2010, the year in which ZFS-On-Linux became available I believe. This is just one of many examples: https://www.phoronix.com/news/MTI1MzM In contrary, in some cases FUSE is even faster than doing a regular kernel mount(). There is an experimental research distribution called distri that is exclusively relying on fuse mounting and figured out that FUSE was fast…

You're conflating zfs-fuse and zfs-on-linux (ZoL).

The first one is as it's name suggests.. a user space implementation of ZFS.

ZoL (now unified with OpenZFS) is implemented as a kernel module and as such does _not_ run in user space. It performs significantly better as a result.

FUSE is still slow, which is why there's ongoing effort to replace things like NTFS-3G (the default NTFS implementation in most linux distros) with an in-kernel implementation: https://news.ycombinator.com/item?id=28418674

Re: LiteFS

#72

Post by hinkley »

pphysch wrote: post #16

> Developing against a relational database requires devs to watch out for "N+1" query patterns, where a query leads to a loop that leads to more queries. N+1 queries against Postgres and MySQL can be lethal to performance. Not so much for SQLite. This is misleading AFAICT. The article(s) is actually comparing remote RDBMS to local RDBMS, not Postgres to SQLite. Postgres can also be served over a UNIX socket, removing…

Yep. The poison is still in the cup, it's just a smaller dose. Keep drinking if you think that fact is enough to save you...

I see bad judgement calls coming from small numbers, often due to failing to do the cost x frequency math properly in your head. If you're looking at an operation that takes 3ms or 3μs that is called a million times per minute, or twenty thousand times per request, you don't have enough significant figures there and people make mistakes. 3ms x 12345 = ~40s, not 37035ms. Better if you use a higher resolution clock and find out it's actually 3.259 ms, leading to a total of ~40.23s

Point is, when we are doing billions of operations per second, lots of small problems that hide in clock jitter can become really big problems, and a 5-10% error repeated for half a dozen concerns can lead to serious miscalculations in capacity planning and revenue.

Re: LiteFS

#73

Post by cbsmith »

lbhdc wrote: post #65
tptacek wrote: post #18

Earlier quoted context omitted.

Anyone want to chime in with the largest app they've deployed where prod Postgres was reachable over a Unix domain socket?

We use managed postgres on GCP, and it is served over unix socket.

Kind of by definition, GCP's managed postgres is served up over the network. I'm guessing you mean "unix socket" in the sense of the socket API, not unix domain sockets.

Re: LiteFS

#75

Post by sk55 »

infogulch wrote: post #47

> To improve latency, we're aiming at a scale-out model that works similarly to Fly Postgres. That's to say: writes get forwarded to the primary and all read requests get served from their local copies. How can you ensure that a client that just performed a forwarded write will be able to read that back on their local replica on subsequent reads?

They don't have ACID guarantees with this setup.

Re: LiteFS

#76

Post by hinkley »

infogulch wrote: post #47

> To improve latency, we're aiming at a scale-out model that works similarly to Fly Postgres. That's to say: writes get forwarded to the primary and all read requests get served from their local copies. How can you ensure that a client that just performed a forwarded write will be able to read that back on their local replica on subsequent reads?

That's a fun one.

A couple years ago someone posted a solution to that here. I'm not sure if it works for SQLite, but it worked for Postgres. The basics of it were that each replica was aware of the latest transaction ID it had seen. On a normal read you'd deal with the usual set of eventually consistent issues. But on a read-after-write, you would select a replica that was ahead of the write transaction.

Ultimately that's a very odd flavor of sharding. What I don't recall is how they propagated that shard information efficiently, since with sharding and consistent hashing the whole idea is that the lookup algorithm is deterministic, and therefore can be run anywhere and get the same result. WAL lag information is dynamic, so... Raft?

Re: LiteFS

#77

Post by wglb »

theomega wrote: post #12

Does anyone else bump into the issue, that the fly.io website does not load if requested via IPv6 on Mac? I tried Safari, Chrome and curl and neither work: $ curl -v https://fly.io/blog/introducing-litefs/ * Trying 2a09:8280:1::a:791:443... * Connected to fly.io (2a09:8280:1::a:791) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/cert.pem…

Just tried it on my linux box:

  * Trying 2a09:8280:1::a:791:443...
  * Connected to fly.io (2a09:8280:1::a:791) port 443 (#0) ALPN,
  * offering h2 ALPN, offering http/1.1 CAfile:
  * /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs TLSv1.0
  * (OUT), TLS header, Certificate Status (22): TLSv1.3 (OUT), TLS
  * handshake, Client hello (1): TLSv1.2 (IN), TLS header, Certificate
  * Status (22): TLSv1.3 (IN), TLS handshake, Server hello (2): TLSv1.2
  * (IN), TLS header, Finished (20): TLSv1.2 (IN), TLS header,
  * Supplemental data (23): TLSv1.3 (IN), TLS handshake, Encrypted
  * Extensions (8): TLSv1.2 (IN), TLS header, Supplemental data (23):
  * TLSv1.3 (IN), TLS handshake, Certificate (11): TLSv1.2 (IN), TLS a
  * header, Supplemental data (23): TLSv1.3 (IN), TLS handshake, CERT
  * verify (15): TLSv1.2 (IN), TLS header, Supplemental data (23):
  * TLSv1.3 (IN), TLS handshake, Finished (20): TLSv1.2 (OUT), TLS
  * header, Finished (20): TLSv1.3 (OUT), TLS change cipher, Change
  * cipher spec (1): TLSv1.2 (OUT), TLS header, Supplemental data (23):
  * TLSv1.3 (OUT), TLS handshake, Finished (20): SSL connection using
  * TLSv1.3 / TLS_AES_256_GCM_SHA384 ALPN, server accepted to use h2
  * Server certificate:
...

Same on my macbook

Re: LiteFS

#78

Post by omnimus »

benbjohnson wrote: post #38
nicoburns wrote: post #37

Where is the data actually being stored in this setup? A copy on each machine running the application? If so, is there another copy somewhere else (e.g. S3) in case all nodes go down? Also, what happens if the Consul instance goes down? If my application nodes can't be ephemeral then this seems like it would be harder to operate than Postgres or MySQL in practice. If it completely abstracts that away somehow then I s…

> Where is the data actually being stored in this setup? A copy on each machine running the application? Yes, each node has a full copy of the database locally. > If so, is there another copy somewhere else (e.g. S3) in case all nodes go down? S3 replication support is coming[1]. Probably in the next month or so. Until then, it's recommended that you run a persistent volume with your nodes. > What happens if the Cons…

Will this replicate the filesystem/volume between instances? Can for example use text files or different kind of filebased database? Or it requires me to use sqlite?

Re: LiteFS

#79

Post by simonw »

cbsmith wrote: post #70
benbjohnson wrote: post #31

Earlier quoted context omitted.

You could run Postgres over UNIX sockets although you will still get higher latency than SQLite's in-process model. Also, running a Postgres on every app instance on the edge probably isn't practical. Postgres has some great advanced features if you need them but it's also much more heavy weight. With LiteFS, we're aiming to easily run on low resource cloud hardware such as nodes with 256MB or less of RAM. I haven't…

Isn't this all talking to S3 anyway, not to mention the network trips intrinsic to the system before it gets to S3? I mean, I'm sure there's some performance win here, but I'm surprised it's so significant. It's not like the address-space separation is without benefits... heck, if it weren't, you could simply have embedded the whole application inside Postgres and achieved the same effect.

You may be confusing Litestream and LiteFS.

Litestream writes everything to S3 (or similar storage).

LiteFS lets different nodes copy replicated data directly to each other over a network, without involving S3.

In either case, the actual SQLite writes and reads all happen directly against local disk, without any network traffic. Replication happens after that.

Re: LiteFS

#80

Post by simonw »

pweissbrod wrote: post #67
pphysch wrote: post #16

> Developing against a relational database requires devs to watch out for "N+1" query patterns, where a query leads to a loop that leads to more queries. N+1 queries against Postgres and MySQL can be lethal to performance. Not so much for SQLite. This is misleading AFAICT. The article(s) is actually comparing remote RDBMS to local RDBMS, not Postgres to SQLite. Postgres can also be served over a UNIX socket, removing…

Select N+1 is a fundamental anti-pattern of relational database usage. Reducing the latency per round trip doesn't change this fact.

I disagree. I think it makes a big difference.

https://sqlite.org/np1queryprob.html#the_need_for_over_200_s... talks about this in the context of Fossil, which uses hundreds of queries per page and loads extremely fast.

It turns out the N+1 thing really is only an anti-pattern if you're dealing with significant overhead per query. If you don't need to worry about that you can write code that's much easier to write and maintain just by putting a few SQL queries in a loop!

Related: the N+1 problem is notorious in GraphQL world as one of the reasons building a high performance GraphQL API is really difficult.

With SQLite you don't have to worry about that! I built https://datasette.io/plugins/datasette-graphql on SQLite and was delighted at how well it can handle deeply nested queries.

Post reply on HN