Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LiteFS Cloud: Distributed SQLite with Managed Backups

fly.io

Post reply on HN
Print view View original

61–70 of 94 posts

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#61

Post by tptacek »

gunapologist99 wrote: post #59

What is the use case for this distributed SQLite compared to a "traditional" distributed database like DynamoDB, CouchDB, Foundation, Riak, TiDB or CockroachDB, Mongo, etc. Is this intended for b2b applications, or b2c? Could you (theoretically) write Facebook with a couple billion users with such a distributed SQLite system (billions of sqlite files, or many many billions of rows in this sort of a system)? I think w…

It's a general approach for SQL-backed CRUD applications. If your application is (1) read-heavy (most are), especially if it's particularly read-heavy, and (2) benefits from snappy responses --- ie, if it's the kind of thing where you'd invest significant time in, like, serverside Javascript compilation --- and, especially, if (3) it wants to run in more than one geographical region at the same time, then LiteFS is an interesting way to slash database server round trip times out of your response budget.

The best way to think about SQLite in modern CRUD apps is by thinking about the N+1 query problem. N+1 is an issue largely because of the round-trips between the app and the database server (in an n-tier architecture, those are virtually always separate servers; in a geographically distributed n-tier architecture, they're also far apart from each other). Think of SQLite as a tool that would allow you to just randomly write N+1 query logic without worrying about it. You'd probably still not do that, but that's the kind of thing SQLite ostensibly lets you get away with.

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#62

Post by tptacek »

lbotos wrote: post #58

How does the billing/pricing for LiteFS and LiteFS Cloud work? I tried to figure it out but it was unclear from docs. https://fly.io/docs/about/pricing/

From the post:

We'll introduce pricing in the coming months, but for now LiteFS Cloud is in preview and is free to use. Please go check it out, and let us know how it goes!

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#63

Post by benbjohnson »

felipeccastro wrote: post #56
benbjohnson wrote: post #48

Earlier quoted context omitted.

I agree with everything the OP said above. Typically if you need to scale writes in SQLite, you'll want to look at sharding. The "single writer" restriction is per database so you can split your SaaS customers across multiple databases. If your SaaS is in the hundreds or thousands of customers then you could split each customer into their own database. That also provides nice tenant isolation. If you have more custom…

Doesn't the WAL mode solve the high concurrency write situation? If it can't be relied on busy season, why the push for Sqlite in production?

WAL solves the high concurrency read situation. Not the writes. SQLite can do thousands of writes per second in WAL mode which is more than enough for the vast majority of applications out there. It's not like most businesses could fulfill thousands of orders per second even if their database could write them.

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#64

Post by lbotos »

tptacek wrote: post #62
lbotos wrote: post #58

How does the billing/pricing for LiteFS and LiteFS Cloud work? I tried to figure it out but it was unclear from docs. https://fly.io/docs/about/pricing/

From the post: We'll introduce pricing in the coming months, but for now LiteFS Cloud is in preview and is free to use. Please go check it out, and let us know how it goes!

<3 Classic HN reader here, I skimmed, went to comments, and then went to pricing :P

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#66

Post by benbjohnson »

yread wrote: post #65

Looks cool. But is LiteFS really ready for production? The github issues page seems to have some lost writes fixed just three weeks ago. I would consider using it instead of running .backup nightly, so having a missed write vs running a cron job is a no brainer.

Author here. We run LiteFS in production for several services internally. The recent issue was pretty rare and it took a while just to be able to reproduce it. There's always going to be bugs in all software. Hopefully they just become increasingly more rare.

As for "production ready", it's tough to define. It's more a question of risk tolerance. I authored a database called BoltDB about ten years ago and there wasn't a point in time where it suddenly crossed into being production ready. At first, it was used for temporary or cached data. Then people used it for derived data like analytics. And eventually it matured enough that it became a commonplace database in the Go ecosystem. It's in tools like etcd which, in turn, is built into systems like Kubernetes.

I expect LiteFS will see similar a similar adoption pattern and eventually move into more and more risk-adverse applications as it continues to mature. LiteFS Cloud aims to help mitigate risk by having streaming backups & point-in-time recovery.

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#67

Post by bjyule »

Spunkie wrote: post #6

I'm really glad to see this complete solution drop. I've cobbling together sqlite point-in-time restores, automatic snapshots, and S3 backups with litestream and scripts for too long. I've been bugging cloudflare to do this with D1+tunnels since D1 was announced and they constantly seemed confused what I was even talking about.

What do you have in mind with respect to D1+Tunnels?

We're launching PITR support for the new experimental backend within the next 1-2 weeks. Stay tuned.

(I am the eng director for Cloudflare Workers)

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#68

Post by gregwebs »

Async replication implemented with 1 second of data loss expected when the primary goes down.

> However, we don't write every individual LTX file to object storage immediately. The latency is too high and it's not cost effective when you write a lot of transactions. Instead, the LiteFS primary node will batch up its changes every second and send a single, compacted LTX file to LiteFS Cloud. Once there, LiteFS Cloud will batch these 1-second files together and flush them to storage periodically.

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#70

Post by maxpert »

I am going to sound like broken record throwing in shameless plugs. But for people who might want to look for alternate options, Marmot (https://github.com/maxpert/marmot) is another option that can let you both replicate, and backup your database. I have recently embedded NATS into Marmot and based on community feedback introduced example script that gets you up and running with example DB within seconds.
Post reply on HN