SQLite is all you need for durable workflows • Hacker News Board

Re: SQLite is all you need for durable workflows

#411

Post by bmurphy1976 » Tue, Jun 02, 2026, 11:38 PM UTC

mnahkies wrote: post #283

kenforthewin wrote: post #184

Earlier quoted context omitted.

Could you expand on the "substantial ops burden"? Let's say you're using a managed Postgres instance as the underlying data store, how substantial is the ops burden in that case? I understand that temporal is actually a set of 4 or so microservices on top of a data store, but if you're already running a distributed system backed by k8s or something like that, it doesn't seem like it adds significant incremental ops o…

In my experience with a relatively modest number of concurrent workflows (think hundreds) you'll be pushing several thousand transactions per second through that postgres instance. As best I can tell it doesn't do any batching of it's writes/reads, and it's update heavy in places rather than append (I suspect their cloud version might do some of these things) It's pretty close to "let's make every function call seria…

At those rates if you aren't sharding your infra you are doing it wrong.

Re: SQLite is all you need for durable workflows

#412

Post by switchbak » Wed, Jun 03, 2026, 9:30 PM UTC

lunar_mycroft wrote: post #410

switchbak wrote: post #408

Earlier quoted context omitted.

Yeah no - sharding SQLite? How about if you know you're going to scale like that, build with something appropriate in the first place.

First, you're very likely underestimating how much load SQLite can handle. SQLite is usually write limited, but for smallish writes it can easily handle thousands a second with very trivial optimization, and with more thought can scale to tens or hundreds of thousands of write transactions per second. In some cases, it can actually out perform traditional server based RDMSs because of reduced overhead and because hol…

I don't care that much about performance - I imagine any modern machine can handle that load pretty well, for this context that's not a huge concern.

The concerns are more about distributed systems issues, maintenance, etc. Typical NIH issues. Which you're embracing when you start building your own systems (ie: with sharding).

This is a solved problem, why would I want to inject these burdens onto myself, instead of solving ... actual problems that I already have enough of! I don't want to be building another Temporal/Cadence/DBOS/etc.

Re: SQLite is all you need for durable workflows

#413

Post by apatheticonion » Mon, Jun 08, 2026, 1:33 PM UTC

Rohansi wrote: post #228

apatheticonion wrote: post #224

Earlier quoted context omitted.

For me, I have a use case that needs to support a few thousand users, probably a few hundred concurrently. The combination of SQLite (libsql, a concurrent implementation of sqlite) and Rust means I can do so from a $2/m VPS and a single server instance. Backups are done via a cron job that uploads to S3. Does it pass the "Netflix scale" test? No But it doesn't need to. I'm not profiting from the service and SQLite of…

> The combination of SQLite (libsql, a concurrent implementation of sqlite) and Rust means I can do so from a $2/m VPS and a single server instance. You can probably do it with regular SQLite, too. Being limited to a single writer isn't as devastating as it sounds when they get processed very quickly. Probably don't need Rust either but it'll be more efficient than the usual choices. (Also, it looks like libsql is th…

Makes sense. I honestly haven't thought about it too much. I liked that I could move to Turso if I needed cloud storage given the free tier and that the API works with tokio nicely