Live data from Hacker News

LiteFS Cloud: Distributed SQLite with Managed Backups

fly.io

11–20 of 94 posts

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#11
post #2

This is awesome. Fly.io is now a good replacement for my small team and some indie projects. Currently on render before but their team pricing turned me off.

Fly gives away a lot for free and yet is cheaper than most NewCloud alternatives. Their patient (and painful) investment in building atop CoLos (instead of BigCloud ) will pay off in the long run, just like it did for Cloudflare. Not sure if Fly already builds their own hardware, though.

Nope. We just buy it.

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#12
post #6

I'm really glad to see this complete solution drop. I've cobbling together sqlite point-in-time restores, automatic snapshots, and S3 backups with litestream and scripts for too long. I've been bugging cloudflare to do this with D1+tunnels since D1 was announced and they constantly seemed confused what I was even talking about.

Let me know how it works for you if you get a chance to try it out. We learned a lot since making Litestream so hopefully it's a big step up! The underlying format of how we're storing snapshots and incremental backups should make restores A LOT faster. Litestream has to replay all WAL writes since the last snapshot whereas LiteFS is able to perform a lot of deduplication and use multi-level compaction to create a minimal set of restore data.

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#13

Do they backup a snapshot? So every 5 minutes for 30 days would be a lot of snapshots to S3

Author here. We take a daily snapshot and then store incremental changes in multiple time intervals to optimize the storage usage and optimize the time to restore. Your app just sends us the incremental changes every second and we handle all the optimization on our side.

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#15

Man, i already intended to try Fly for my upcoming hosting needs. Ya'll keep making the pot sweeter, though. I'm really curious to see how some of these SQLite toolings will work in the "dumb and simple app" case. Ie i'm writing an app that is focused on being local, single instance. Which i know is blasphemy to Fly, but it's my target audience - self hosting first and foremost. I had planned on trying Fly through th…

Author here. I love single-instance deployments. I think they make a lot of sense when you don't need high-availability. We have quite a number of people that run Litestream on a single instance with a volume attached. We run Litestream internally in several places as well.

LiteFS tries to improve upon Litestream by making it simple to integrate backups and make it easy to scale out your application in different regions without changing your app. I don't think every application needs to be on the edge but we're hoping to make it easy enough that it's more a question of "why not?"

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#17
Very similar to mvsqlite (also has snapshots with its own MVCC). Sadly you cannot self host this, though. It's true you can open source LiteFS, but this seems to have additional features.

I still think the replicated sqlite approach is the wrong one, though. It makes sense for fly, but most people don't need replication, they need sharding. The parallel write issue with SQLite remains sadly.

Ideally you would have both - a cluster (let's say 3 instances) for SQLite, in which writes a committed transactionally, and then LiteFS, where you shard on a key (let's say in the instnace of HN, by thread) and create separate SQLite DB for each where you get (stale) fast reads.

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#18
What are the reasons to continue using SQLite over MySQL/MariaDB when you start to require distributed architectures? Wouldn't it be better to switch at that point? Assuming that being able to read from a database on the same filesystem as the application doesn't provide any tangible benefits for 99.99% of applications that don't have such low latency requirements?

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#19

Man, i already intended to try Fly for my upcoming hosting needs. Ya'll keep making the pot sweeter, though. I'm really curious to see how some of these SQLite toolings will work in the "dumb and simple app" case. Ie i'm writing an app that is focused on being local, single instance. Which i know is blasphemy to Fly, but it's my target audience - self hosting first and foremost. I had planned on trying Fly through th…

Author here. I love single-instance deployments. I think they make a lot of sense when you don't need high-availability. We have quite a number of people that run Litestream on a single instance with a volume attached. We run Litestream internally in several places as well. LiteFS tries to improve upon Litestream by making it simple to integrate backups and make it easy to scale out your application in different regi…

Hey Ben, great to see this progress!

Is it a goal of LiteFS to serve single-instance deployments as well as Litestream does? Would you say LiteFS has already achieved that at this point, or would Litestream still be the better match for single-instance apps?

I've experimented with LiteFS and liked it, but all my apps are single-deployment, so I've stuck with Litestream. But I know LiteFS is receiving much more investment, so I'm wondering if Litestream is long for this world.

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#20
post #18

What are the reasons to continue using SQLite over MySQL/MariaDB when you start to require distributed architectures? Wouldn't it be better to switch at that point? Assuming that being able to read from a database on the same filesystem as the application doesn't provide any tangible benefits for 99.99% of applications that don't have such low latency requirements?

There's an operational argument to make, but fundamentally it's about performance, and secondarily about what having an ultra-fast local database does to your performance budget and thus how you build things. The premise is that in a typical web application, each request incurs multiple round trips to the database, and those round trips add up to eat a chunk of your budget. A database like SQLite can essentially eliminate that round-trip cost.

It's not just about making snappier applications; it's also that even ordinary apps burn engineering time (for most shops, the most expensive resource) on minimizing those database round trips --- it's why so much ink has been spilt about hunting and eliminating N+1 query patterns, for instance, which is work you more or less don't have to think about with SQLite.

This premise doesn't hold for all applications, or maybe even most apps! But there is a big class of read-heavy applications where it's a natural fit.

Post reply on HN