Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LiteFS Cloud: Distributed SQLite with Managed Backups

fly.io

Post reply on HN
Print view View original

21–30 of 94 posts

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#21

Post by tptacek »

endisneigh wrote: post #17

Very similar to mvsqlite (also has snapshots with its own MVCC). Sadly you cannot self host this, though. It's true you can open source LiteFS, but this seems to have additional features. I still think the replicated sqlite approach is the wrong one, though. It makes sense for fly, but most people don't need replication, they need sharding. The parallel write issue with SQLite remains sadly. Ideally you would have bo…

Why do most people need sharding? Most people have a single RDS instance that they scale vertically. Most people aren't pushing the limits of vertical scaling, and LiteFS isn't a "horizontal" scaling alternative to conventional n-tier SQL systems.

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#23

Post by benbjohnson »

stu_k wrote: post #22

Has anyone built a mobile app on top of SQLite that can work offline, but then sync to a server when it gets connectivity? It feels like this could be built with a similar approach to this distributed SQLite, you'd "just" need more robust conflict handling.

I haven't personally but typically people use a CRDT-based approach like vlcn[1] or Mycelite[2].

[1]: https://github.com/vlcn-io/cr-sqlite

[2]: https://github.com/mycelial/mycelite

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#24

Post by benbjohnson »

mtlynch wrote: post #19
benbjohnson wrote: post #15

Earlier quoted context omitted.

Author here. I love single-instance deployments. I think they make a lot of sense when you don't need high-availability. We have quite a number of people that run Litestream on a single instance with a volume attached. We run Litestream internally in several places as well. LiteFS tries to improve upon Litestream by making it simple to integrate backups and make it easy to scale out your application in different regi…

Hey Ben, great to see this progress! Is it a goal of LiteFS to serve single-instance deployments as well as Litestream does? Would you say LiteFS has already achieved that at this point, or would Litestream still be the better match for single-instance apps? I've experimented with LiteFS and liked it, but all my apps are single-deployment, so I've stuck with Litestream. But I know LiteFS is receiving much more invest…

Thanks, Michael! Before LiteFS Cloud, I would have said that LiteFS doesn't add much value over vanilla SQLite for single-node deployments. But with the new streaming backup feature, I think it makes it easier to run single-node applications than Litestream. There's some additional benefits like being able to import/export the database remotely and doing live point-in-time restores. The main drawback to LiteFS compared to Litestream is write performance but we're alleviating that by implementing a VFS option in the future.

We do have some nice features coming down the pipe for LiteFS so you can use it with purely ephemeral nodes. Let me know if you get a chance to try it out and if there's any improvements you'd like to see.

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#25

Post by FranGro78 »

tptacek wrote: post #21
endisneigh wrote: post #17

Very similar to mvsqlite (also has snapshots with its own MVCC). Sadly you cannot self host this, though. It's true you can open source LiteFS, but this seems to have additional features. I still think the replicated sqlite approach is the wrong one, though. It makes sense for fly, but most people don't need replication, they need sharding. The parallel write issue with SQLite remains sadly. Ideally you would have bo…

Why do most people need sharding? Most people have a single RDS instance that they scale vertically. Most people aren't pushing the limits of vertical scaling, and LiteFS isn't a "horizontal" scaling alternative to conventional n-tier SQL systems.

Definitely not a concern for “most” folks, but I’ve seen sharding used to separate users into smaller pools which do not have shared fate wrt infra and deployments.

As an extra benefit these pools do not push the limits of vertical scaling to which makes them both easier and cheaper to test under high load conditions.

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#26

Post by psanford »

Man this is cool. While I really enjoy my own solution of using a custom SQLite vfs that stores your db transparently in dynamodb[0], this really is a compelling alternative.

I wonder how viable this would be to use from aws lambda? It seems like the way lambda does concurrency probably doesn't play all that well with litefs. Maybe it's time to move some workloads over to fly.io.

[0]: https://github.com/psanford/donutdb

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#27

Post by qweqwe14 »

stu_k wrote: post #22

Has anyone built a mobile app on top of SQLite that can work offline, but then sync to a server when it gets connectivity? It feels like this could be built with a similar approach to this distributed SQLite, you'd "just" need more robust conflict handling.

Not 100% sure if it uses SQLite, but there's an app called "Flaming Durtles" on Android. It's a frontend for WaniKani (Japanese learning app). The app downloads all the lessons from WK's servers, and you can do them even when offline. When you go online, it uploads them to the backend and syncs everything else. In my experience, it works flawlessly.

It's actually a 3rd party app, WK provides an API so that people can make apps that interface with the system.

Forum thread: https://community.wanikani.com/t/android-flaming-durtles-and...

Source code (may be outdated): https://github.com/ejplugge/com.the_tinkering.wk

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#29

Post by crop_rotation »

HN seems to love sqlite but I don't know how to protect against dataloss using sqlite (yes I know litestream exists). Unless you sync disk state to remote storage on every write, wouldn't you always lose some writes? Losing writes is never a good idea. You might think it doesn't matter at your scale, but suddenly it might and you won't like SQLite the same amount anymore.

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#30

Post by mtlynch »

benbjohnson wrote: post #24
mtlynch wrote: post #19

Earlier quoted context omitted.

Hey Ben, great to see this progress! Is it a goal of LiteFS to serve single-instance deployments as well as Litestream does? Would you say LiteFS has already achieved that at this point, or would Litestream still be the better match for single-instance apps? I've experimented with LiteFS and liked it, but all my apps are single-deployment, so I've stuck with Litestream. But I know LiteFS is receiving much more invest…

Thanks, Michael! Before LiteFS Cloud, I would have said that LiteFS doesn't add much value over vanilla SQLite for single-node deployments. But with the new streaming backup feature, I think it makes it easier to run single-node applications than Litestream. There's some additional benefits like being able to import/export the database remotely and doing live point-in-time restores. The main drawback to LiteFS compar…

Cool, I'll give it a spin. Thanks, Ben!
Post reply on HN