Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LiteFS

fly.io

Post reply on HN
Print view View original

31–40 of 158 posts

Re: LiteFS

#31

Post by benbjohnson »

pphysch wrote: post #16

> Developing against a relational database requires devs to watch out for "N+1" query patterns, where a query leads to a loop that leads to more queries. N+1 queries against Postgres and MySQL can be lethal to performance. Not so much for SQLite. This is misleading AFAICT. The article(s) is actually comparing remote RDBMS to local RDBMS, not Postgres to SQLite. Postgres can also be served over a UNIX socket, removing…

You could run Postgres over UNIX sockets although you will still get higher latency than SQLite's in-process model. Also, running a Postgres on every app instance on the edge probably isn't practical. Postgres has some great advanced features if you need them but it's also much more heavy weight.

With LiteFS, we're aiming to easily run on low resource cloud hardware such as nodes with 256MB or less of RAM. I haven't tried it on a Raspberry Pi yet but I suspect it would run fine there as well.

Re: LiteFS

#32

Post by dcre »

whoisjuan wrote: post #22

Unrelated, but why does the map on their homepage show a region in Cuba? That must be wrong.

I think it's in Miami and the icon is confusingly centered over it. The icons look like pins but I think they're supposed to be hot-air balloons. This explains why some of them appear to point to spots in the ocean.

https://fly.io/docs/reference/regions/

Re: LiteFS

#33

Post by tptacek »

whalesalad wrote: post #28
tptacek wrote: post #18

Earlier quoted context omitted.

Anyone want to chime in with the largest app they've deployed where prod Postgres was reachable over a Unix domain socket?

is it cheating when pgbouncer is being accessed via socket? lol

Yes, that is cheating.

Re: LiteFS

#34

Post by mgkimsal »

pphysch wrote: post #29
woooooo wrote: post #23

Earlier quoted context omitted.

Isn't "n + 1" typically a degenerate case of ORM rather than a relational thing? The whole point of rdbms is you can do a join instead of iterated lookups.

Yeah most N+1 issues can be mitigated by optimizing the queries. You definitely don't need an ORM to get to N+1 hell, a for-loop in vanilla PHP will do.

A for-loop in vanilla anything will do. Nothing special about PHP. Ruby, Python, Java, etc...

Re: LiteFS

#35

Post by pphysch »

benbjohnson wrote: post #31
pphysch wrote: post #16

> Developing against a relational database requires devs to watch out for "N+1" query patterns, where a query leads to a loop that leads to more queries. N+1 queries against Postgres and MySQL can be lethal to performance. Not so much for SQLite. This is misleading AFAICT. The article(s) is actually comparing remote RDBMS to local RDBMS, not Postgres to SQLite. Postgres can also be served over a UNIX socket, removing…

You could run Postgres over UNIX sockets although you will still get higher latency than SQLite's in-process model. Also, running a Postgres on every app instance on the edge probably isn't practical. Postgres has some great advanced features if you need them but it's also much more heavy weight. With LiteFS, we're aiming to easily run on low resource cloud hardware such as nodes with 256MB or less of RAM. I haven't…

SQLite almost certainly is the better edge RDBMS than Postgres, if only because it has less features taking up space.

However, "local SQLite vs. remote Postgres/MySQL" remains a false dichotomy when talking about network latency.

Re: LiteFS

#36

Post by pphysch »

mgkimsal wrote: post #34
pphysch wrote: post #29

Earlier quoted context omitted.

Yeah most N+1 issues can be mitigated by optimizing the queries. You definitely don't need an ORM to get to N+1 hell, a for-loop in vanilla PHP will do.

A for-loop in vanilla anything will do. Nothing special about PHP. Ruby, Python, Java, etc...

Of course. PHP is the one you will mainly encounter in the wilderness, though.

Re: LiteFS

#37

Post by nicoburns »

Where is the data actually being stored in this setup? A copy on each machine running the application? If so, is there another copy somewhere else (e.g. S3) in case all nodes go down?

Also, what happens if the Consul instance goes down?

If my application nodes can't be ephemeral then this seems like it would be harder to operate than Postgres or MySQL in practice. If it completely abstracts that away somehow then I suppose that'd be pretty cool.

Currently finding it hard to get on board with the idea that adding a distributed system here actually makes things simpler.

Re: LiteFS

#38

Post by benbjohnson »

nicoburns wrote: post #37

Where is the data actually being stored in this setup? A copy on each machine running the application? If so, is there another copy somewhere else (e.g. S3) in case all nodes go down? Also, what happens if the Consul instance goes down? If my application nodes can't be ephemeral then this seems like it would be harder to operate than Postgres or MySQL in practice. If it completely abstracts that away somehow then I s…

> Where is the data actually being stored in this setup? A copy on each machine running the application?

Yes, each node has a full copy of the database locally.

> If so, is there another copy somewhere else (e.g. S3) in case all nodes go down?

S3 replication support is coming[1]. Probably in the next month or so. Until then, it's recommended that you run a persistent volume with your nodes.

> What happens if the Consul instance goes down?

If Consul goes down then you'll lose write availability but your nodes will still be able to perform read queries.

> If my application nodes can't be ephemeral then this seems like it would be harder to operate than Postgres or MySQL in practice.

Support for pure ephemeral nodes is coming. If you're comparing LiteFS to a single node Postgres/MySQL then you're right, it's harder to operate. However, distributing out Postgres/MySQL to regions around the world and handling automatic failovers is likely more difficult to operate than LiteFS.

Re: LiteFS

#39

Post by nucleardog »

tptacek wrote: post #18
pphysch wrote: post #16

> Developing against a relational database requires devs to watch out for "N+1" query patterns, where a query leads to a loop that leads to more queries. N+1 queries against Postgres and MySQL can be lethal to performance. Not so much for SQLite. This is misleading AFAICT. The article(s) is actually comparing remote RDBMS to local RDBMS, not Postgres to SQLite. Postgres can also be served over a UNIX socket, removing…

Anyone want to chime in with the largest app they've deployed where prod Postgres was reachable over a Unix domain socket?

Used to work developing standalone kiosks and things (think along the lines of stuff you'd find at a museum as the basic type of thing I'd work on), so absolutely not the use case you're thinking of.

In a number of cases, we were working with data that would really benefit from actual GIS tooling so PostGIS was kind of the natural thing to reach for. Many of these kiosks had slow, intermittent, or just straight up no internet connection available.

So we just deployed PostGIS directly on the kiosk hardware. Usually little embedded industrial machines with passive cooling working with something like a generation or two out-of-date Celeron, 8GB RAM, and a small SSD.

We'd load up shapefiles covering a bunch of features across over a quarter million square miles while simultaneously running Chromium in kiosk mode.

I know for a fact some of those had around 1k DAU. I mean, never more than one simultaneously but we did have them! I'm sure it would have handled more just fine if it weren't for the damn laws of physics limiting the number of people that can be physically accessing the hardware at the same time.

That said, we had the benefit of knowing that our user counts would never really increase and due to real-world limitations we'd never be working with a remote database because there was literally no internet available. In general I'd still say colocating your database is not an excuse to allow N+1 query patterns to slip in. They'll be fine for now and just come back to bite you in the ass when your app _does_ outgrow colocating the app/database.

Re: LiteFS

#40

Post by dspillett »

woooooo wrote: post #23
pphysch wrote: post #16

> Developing against a relational database requires devs to watch out for "N+1" query patterns, where a query leads to a loop that leads to more queries. N+1 queries against Postgres and MySQL can be lethal to performance. Not so much for SQLite. This is misleading AFAICT. The article(s) is actually comparing remote RDBMS to local RDBMS, not Postgres to SQLite. Postgres can also be served over a UNIX socket, removing…

Isn't "n + 1" typically a degenerate case of ORM rather than a relational thing? The whole point of rdbms is you can do a join instead of iterated lookups.

You see it in direct code too, from new or naive coders (the good old "it was fast enough on my test data…") or when proof-of-concept code "accidentally" escapes into production.

Even within the database you'll sometimes find examples: stored procedures looping over a cursor and running queries or other statements in each iteration. This doesn't have the same latency implications as N+1 requests over a network, but still does add significant time per step compared to a set based equivalent.

If you think about it, the database will sometimes do this itself when your database is not optimally structured & indexed for your queries (in SQL Server, look at "number of executions" in query plans).

Post reply on HN