Live data from Hacker News

LiteFS

fly.io

11–20 of 158 posts

Re: LiteFS

#11
post #4

This is distributed SQLite 3, running (I assume at least partially managed?) LiteFS[5] for you. Which is pretty cool! What I'd like to have seen is how this compares to things like rqlite[1] or Cloudflare's D1[2] addressed directly in the article That said, I think this is pretty good for things like read replica's. I know the sales pitch here is as a full database, and I don't disagree with it, and if I was starting…

> This is distributed SQLite 3, running (I assume at least partially managed?) Litestream[0] for you. Litestream and LiteFS are by the same author and serve different purposes. https://litestream.io/alternatives/ goes into this. It includes LiteFS and rqlite. [Edited for clarification]

I did see this, its a little light on details in the way I was thinking about this, however I appreciate the call out, especially for others!

It is missing the D1 comparison though.

Re: LiteFS

#12
Does anyone else bump into the issue, that the fly.io website does not load if requested via IPv6 on Mac? I tried Safari, Chrome and curl and neither work:

  $ curl -v https://fly.io/blog/introducing-litefs/
  *   Trying 2a09:8280:1::a:791:443...
  * Connected to fly.io (2a09:8280:1::a:791) port 443 (#0)
  * ALPN, offering h2
  * ALPN, offering http/1.1
  * successfully set certificate verify locations:
  *  CAfile: /etc/ssl/cert.pem
  *  CApath: none
  * (304) (OUT), TLS handshake, Client hello (1):
  curl: (35) error:02FFF036:system library:func(4095):Connection reset by peer

Requesting via ipv4 works

  $ curl -4v https://fly.io/blog/introducing-litefs/
  *   Trying 37.16.18.81:443...
  * Connected to fly.io (37.16.18.81) port 443 (#0)
  * ALPN, offering h2
  * ALPN, offering http/1.1
  * successfully set certificate verify locations:
  *  CAfile: /etc/ssl/cert.pem
  *  CApath: none
  * (304) (OUT), TLS handshake, Client hello (1):
  * (304) (IN), TLS handshake, Server hello (2):
  * (304) (IN), TLS handshake, Unknown (8):
  * (304) (IN), TLS handshake, Certificate (11):
  * (304) (IN), TLS handshake, CERT verify (15):
  * (304) (IN), TLS handshake, Finished (20):
  * (304) (OUT), TLS handshake, Finished (20):
  * SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
  * ALPN, server accepted to use h2
  * Server certificate:
  *  subject: CN=fly.io
  *  start date: Jul 25 11:20:01 2022 GMT
  *  expire date: Oct 23 11:20:00 2022 GMT
  *  subjectAltName: host "fly.io" matched cert's "fly.io"
  *  issuer: C=US; O=Let's Encrypt; CN=R3
  *  SSL certificate verify ok.
  * Using HTTP2, server supports multiplexing
  * Connection state changed (HTTP/2 confirmed)
  * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  * Using Stream ID: 1 (easy handle 0x135011c00)
  > GET /blog/introducing-litefs/ HTTP/2
  > Host: fly.io
  > user-agent: curl/7.79.1
  > accept: */*
  >
  * Connection state changed (MAX_CONCURRENT_STREAMS == 32)!
   ...

Re: LiteFS

#14

Earlier quoted context omitted.

LiteFS/Litestream author here. You bring up a lot of great points that I'll try to address. > What I'd like to have seen is how this compares to things like rqlite or Cloudflare's D1 addressed directly in the article I think a post comparing the different options is a great idea. I'll try to summarize a bit here though. LiteFS aims to be an analogue to Postgres replication but with built-in failover. Postgres uses lo…

Note: I apologize if this is overstepping, its hard to tell! I think a strong - extremely strong - selling point is the point I made about "prebaked" data for your APIs, since the entire strength of these SQLite based systems reside in their fast read capacity (as mentioned elsewhere and in this article, its very fast for read heavy applications, which is most) you could take on an angle around that to get people "in…

> I think a strong - extremely strong - selling point is the point I made about "prebaked" data for your APIs.

I think caches are an excellent use case for LiteFS early on. Sorry I didn't make that point in my previous reply. It's a good way to get benefits out of LiteFS without committing to it as your source of truth. Also related, Segment built a custom SQLite-based solution[1] for distributing out cached data that worked well for them.

[1]: https://segment.com/blog/separating-our-data-and-control-pla...

> We found it was cheaper - by a good margin - to do this over caching everything AOT in a redis cluster

Do you remember specifics of the cost difference? I can imagine it'd be pretty significant since you don't need to spin up servers with a bunch of RAM.

> Note: I apologize if this is overstepping, its hard to tell!

Not overstepping at all! It's great to hear folks' feedback.

Re: LiteFS

#15
post #4

Earlier quoted context omitted.

> This is distributed SQLite 3, running (I assume at least partially managed?) Litestream[0] for you. Litestream and LiteFS are by the same author and serve different purposes. https://litestream.io/alternatives/ goes into this. It includes LiteFS and rqlite. [Edited for clarification]

I did see this, its a little light on details in the way I was thinking about this, however I appreciate the call out, especially for others! It is missing the D1 comparison though.

I'm happy to add D1 onto the alternatives list once it's released. I believe it's still in closed beta though.

Re: LiteFS

#16
> Developing against a relational database requires devs to watch out for "N+1" query patterns, where a query leads to a loop that leads to more queries. N+1 queries against Postgres and MySQL can be lethal to performance. Not so much for SQLite.

This is misleading AFAICT. The article(s) is actually comparing remote RDBMS to local RDBMS, not Postgres to SQLite.

Postgres can also be served over a UNIX socket, removing the individual query overhead due to TCP roundtrip.

SQLite is a great technology, but keep in mind that you can also deploy Postgres right next to your app as well. If your app is something like a company backend that could evolve a lot and benefit from Postgres's advanced features, this may be the right choice.

Re: LiteFS

#17
Well that was fast... [1]

Are the readable replicas supposed to be long-lived (as in, I don't know, hours)? Or does consul happily converge even with ephemeral instances coming and going every few minutes (thinking of something like Cloud Run and the like, not sure if Fly works the same way)? And do they need to make a copy of the entire DB when they "boot" or do they stream pages in on demand?

[1] https://news.ycombinator.com/item?id=32240230

Re: LiteFS

#18
post #16

> Developing against a relational database requires devs to watch out for "N+1" query patterns, where a query leads to a loop that leads to more queries. N+1 queries against Postgres and MySQL can be lethal to performance. Not so much for SQLite. This is misleading AFAICT. The article(s) is actually comparing remote RDBMS to local RDBMS, not Postgres to SQLite. Postgres can also be served over a UNIX socket, removing…

Anyone want to chime in with the largest app they've deployed where prod Postgres was reachable over a Unix domain socket?

Re: LiteFS

#19

ELI5?

LiteFS makes it so that you can have a SQLite database that is transparently replicated to a cluster of machines. The use case is if you have your app running on a server running in Chicago, then users in Europe will have 100ms latency to your server and users in Asia will have a 250ms latency. That's on top of the time it takes for your server to process the request. Many people target sub-100ms latency to make their applications feel snappy but that's impossible if users are waiting on their request to be sent halfway around the world.

Traditionally, it's complicated to replicate your database to different regions of the world using something like Postgres. However, LiteFS aims to make it as simple as just spinning up more server instances around the world. It connects them automatically and ships changes in a transactionally safe way between them.

Re: LiteFS

#20

ELI5?

It’s a tool to allow you to keep your database on the app server and backed up on s3. This means you don’t need a separate database server. Having a database on the same machine as the app is very fast, but it is usually risky. The backups made by LiteFS lessen the risk.
Post reply on HN