Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LiteFS

fly.io

Post reply on HN
Print view View original

51–60 of 158 posts

Re: LiteFS

#51

Post by alberth »

pphysch wrote: post #16

> Developing against a relational database requires devs to watch out for "N+1" query patterns, where a query leads to a loop that leads to more queries. N+1 queries against Postgres and MySQL can be lethal to performance. Not so much for SQLite. This is misleading AFAICT. The article(s) is actually comparing remote RDBMS to local RDBMS, not Postgres to SQLite. Postgres can also be served over a UNIX socket, removing…

A better comparison might be Postgres to BedrockDB.

BeckrockDB is a MySQL compatible interface wrapped around SQLite and as such, is intended for remote RDBMS use.

Expensify has some interesting scaling performance data for their use of it.

https://blog.expensify.com/2018/01/08/scaling-sqlite-to-4m-q...

Re: LiteFS

#52

Post by miduil »

mwcampbell wrote: post #48

I wonder if using FUSE has had any appreciable impact on performance, particularly read performance. I ask because FUSE has historically had a reputation for being slow, e.g. with the old FUSE port of ZFS.

I think FUSE performance has significantly improved since 2010, the year in which ZFS-On-Linux became available I believe.

This is just one of many examples: https://www.phoronix.com/news/MTI1MzM

In contrary, in some cases FUSE is even faster than doing a regular kernel mount(). There is an experimental research distribution called distri that is exclusively relying on fuse mounting and figured out that FUSE was faster for doing mounts. https://michael.stapelberg.ch/posts/tags/distri/

Re: LiteFS

#54

Post by fdr »

tptacek wrote: post #18
pphysch wrote: post #16

> Developing against a relational database requires devs to watch out for "N+1" query patterns, where a query leads to a loop that leads to more queries. N+1 queries against Postgres and MySQL can be lethal to performance. Not so much for SQLite. This is misleading AFAICT. The article(s) is actually comparing remote RDBMS to local RDBMS, not Postgres to SQLite. Postgres can also be served over a UNIX socket, removing…

Anyone want to chime in with the largest app they've deployed where prod Postgres was reachable over a Unix domain socket?

Not me personally, but my understanding some Cisco products have Postgres "embedded." If they're not using unix domain sockets, they're at least using loopback.

...come to think of it, they may not be the only network device maker that does this.

Re: LiteFS

#55

Post by vcryan »

Currently, I am running multiple application servers (and lambda functions) using AWS Fargate that access sqlite files (databases) on an EFS share. So far so good, although my use cases are fairly simple.

Re: LiteFS

#56

Post by jzelinskie »

tptacek wrote: post #18
pphysch wrote: post #16

> Developing against a relational database requires devs to watch out for "N+1" query patterns, where a query leads to a loop that leads to more queries. N+1 queries against Postgres and MySQL can be lethal to performance. Not so much for SQLite. This is misleading AFAICT. The article(s) is actually comparing remote RDBMS to local RDBMS, not Postgres to SQLite. Postgres can also be served over a UNIX socket, removing…

Anyone want to chime in with the largest app they've deployed where prod Postgres was reachable over a Unix domain socket?

I'm surprised no one is chiming in from the LAMP era. Maybe I was just in a bubble, but I feel like before modern cloud providers, folks deploying on VPSs tried to keep operational complexity lower by running their database on the same box until they couldn't anymore.

In my experience, I just cached the living hell out of the apps to avoid the I/O from the database. Providers in those days had huge issues with noisy neighbors, so I/O performance was quite poor.

[0]: https://en.wikipedia.org/wiki/LAMP_(software_bundle)

Re: LiteFS

#57

Post by RussianCow »

woooooo wrote: post #42
pphysch wrote: post #29

Earlier quoted context omitted.

Yeah most N+1 issues can be mitigated by optimizing the queries. You definitely don't need an ORM to get to N+1 hell, a for-loop in vanilla PHP will do.

Yeah, but you have to go out of your way to do that. If you're in touch with the SQL you're executing at all, the join is the obvious easy way to do it, it's not some arcane optimization.

That’s a really big “if”. In my experience, most application engineers aren’t very good at SQL because they don’t have to be; most apps just don't ever reach a scale where it matters, and something like Rails or Django with N+1 queries all over the place performs just fine.

Re: LiteFS

#58

Post by theomega »

toast0 wrote: post #27
theomega wrote: post #12

Does anyone else bump into the issue, that the fly.io website does not load if requested via IPv6 on Mac? I tried Safari, Chrome and curl and neither work: $ curl -v https://fly.io/blog/introducing-litefs/ * Trying 2a09:8280:1::a:791:443... * Connected to fly.io (2a09:8280:1::a:791) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/cert.pem…

Could easily be a path MTU issue? I don't currently have working IPv6, but on TCP/IPv4, the server->client handshake packets I get back from fly.io are full length packets. If your tcpv6 MSS is set incorrectly, and some other not great things are happening in the path, then you might not be able to receive large packets.

How could I find out if this is the case? And, what can I do about it? Disabling IPv6 obviously fixes it, but that it not a solution...

Re: LiteFS

#59

Post by vcryan »

This approach is very appealing to me :) curious about how people handle schema migrations when using this approach.

I segment sqlite files (databases) that have the same schema into the same folder. I haven't really had a case where migrations was really a concern, but I could see it happening soon.

Seems like in my deployment, I'm going to need an approach to loop over dbs to apply this change... I currently have a step of app deployment that attempts to apply migrations... but it is more simplistic because the primary RDBMS (postgresql) just appears to the application as a single entity which is the normative use-case for db-migrate-runners.

Re: LiteFS

#60

Post by woooooo »

RussianCow wrote: post #57
woooooo wrote: post #42

Earlier quoted context omitted.

Yeah, but you have to go out of your way to do that. If you're in touch with the SQL you're executing at all, the join is the obvious easy way to do it, it's not some arcane optimization.

That’s a really big “if”. In my experience, most application engineers aren’t very good at SQL because they don’t have to be; most apps just don't ever reach a scale where it matters, and something like Rails or Django with N+1 queries all over the place performs just fine.

Hence my 2 decade losing battle against ORM at every place I've worked :)

I'm, like, 1 for 10 maybe.

Post reply on HN