SQLite is all you need for durable workflows • Hacker News Board

Re: SQLite is all you need for durable workflows

#221

Post by sieabahlpark » Sat, May 30, 2026, 2:33 AM UTC

rpdillon wrote: post #76

ksd482 wrote: post #68

Earlier quoted context omitted.

levkk is talking about concurrency. The list you gave doesn't explain high concurrency requirements for usage.

My read is that levkk is conflating concurrency with "real production apps" and this whole thread is starting to surface that "real production apps" and "high concurrency" are not measuring the same thing at all. Sqlite is used in real production apps more than any other database. Sqlite is also weak at any sort of write concurrency. Both can be true.

[dead]

Re: SQLite is all you need for durable workflows

#222

Post by hedgehog » Sat, May 30, 2026, 2:34 AM UTC

peterson_lock wrote: post #38

bitexploder wrote: post #7

I started setting up my workflows using Temporal. It deploys as relatively light weight local app. For an isolated local installation it uses SQLite. It makes the process of dealing with API retries and organizing workflows and tasks really simple. I recommend giving it a try. It is, philosophically, exactly what this article is suggesting, but it adds an incredibly rich and flexible interface for agents to work with…

This reads like an advertisement for Temporal :)

It does, my experience has been that it adds code complexity, deployment complexity, and performance problems. There are some observability benefits, but other ways to solve that. It's possible there are workloads that fit it but not anything I've personally worked on.

Re: SQLite is all you need for durable workflows

#223

Post by __turbobrew__ » Sat, May 30, 2026, 2:36 AM UTC

kenforthewin wrote: post #184

svara wrote: post #113

Earlier quoted context omitted.

Word on HN is that you're either paying more money than you expected for temporal's managed solution or taking on substantial ops burden ultimately running their very heavy system yourself. I wouldn't know, I've not done either, but I'd like to learn more from your or other's experience.

Could you expand on the "substantial ops burden"? Let's say you're using a managed Postgres instance as the underlying data store, how substantial is the ops burden in that case? I understand that temporal is actually a set of 4 or so microservices on top of a data store, but if you're already running a distributed system backed by k8s or something like that, it doesn't seem like it adds significant incremental ops o…

I think it depends a lot on the operational maturity of the company. Some places are running the LGTM observability stack, sentry for error reporting, 24/7 on call rotations, playbooks for all alerts, etc. Those organizations will have less issues running systems like temporal because the operational framework is already there.

Other orgs have never heard of alerts or error reporting and naturally will not catch issues until they are catastrophic (for example services that crash frequently in the background go unnoticed until the crash frequency causes a catastrophic failure). In my experience a lot of issues are pretty simple such as running out of memory, CPU throttling, crashes caused by simple bugs (nil panics). If you have good observability you can catch those issues early.

For example: people rag on Ceph that their cluster somehow got into a broken state, but that really only occurs when abuse of the ceph cluster has went on long enough that the cluster finally reaches the tipping point where it is unrecoverable. If you set ceph up, follow the correct replication rules so components are spread across failure domains, and use the metrics and alerts that are distributed with ceph it is actually quite hard to break the cluster.

Re: SQLite is all you need for durable workflows

#224

Post by apatheticonion » Sat, May 30, 2026, 2:45 AM UTC

levkk wrote: post #22

I don't understand this obsession with SQLite for real, production apps. SQLite is an embedded database, completely unsuitable for managing concurrency. This is what database _servers_ are for, e.g., Postgres, MySQL, etc. Their entire job is to allow you to modify data from multiple processes, on different machines, at the same time. This is a foundational principle of computer science. It seems to me that the "SQLit…

For me, I have a use case that needs to support a few thousand users, probably a few hundred concurrently.

The combination of SQLite (libsql, a concurrent implementation of sqlite) and Rust means I can do so from a $2/m VPS and a single server instance.

Backups are done via a cron job that uploads to S3.

Does it pass the "Netflix scale" test? No

But it doesn't need to. I'm not profiting from the service and SQLite offers a path to scale if/when ready because... well it's just SQL and I can literally just swap `libsql::Connection` with `psql::Connection` in my repositories.

Plus upgrading from a $2/m VPS to a $10/month VPS quadripples the number of concurrent users I can support.

IMO, you can vertically scale extraordinary far with SQlite and an efficient server implementation.

I'd wager that 90% of forum websites, wordpress sites and online shops would be fine with SQLite.

Re: SQLite is all you need for durable workflows

#225

Post by dnnddidiej » Sat, May 30, 2026, 2:53 AM UTC

Butchering the Beatles song again.

Re: SQLite is all you need for durable workflows

#226

Post by jrockway » Sat, May 30, 2026, 3:19 AM UTC

levkk wrote: post #22

I don't understand this obsession with SQLite for real, production apps. SQLite is an embedded database, completely unsuitable for managing concurrency. This is what database _servers_ are for, e.g., Postgres, MySQL, etc. Their entire job is to allow you to modify data from multiple processes, on different machines, at the same time. This is a foundational principle of computer science. It seems to me that the "SQLit…

There is something appealing about "it's just a file" (it really isn't; it has locks and a WAL), but I agree with you.

I think people are afraid to read the documentation for postgres. You can start it up in milliseconds. Fast enough and light enough to run one copy for every test case in your test suite, or whatever you're using it for. (mkdir /tmp/whatever; initdb -D /tmp/whatever --no-instructions -A reject -c listen_addresses= --auth-local=trust --no-sync -c fsync=off -c unix_socket_directories=/tmp/whatever -U postgres --no-locale; postgres -D /tmp/whatever) Now you have a test database that behaves exactly like production because it's exactly like production. (OK, turning fsync off makes it a lot faster than production, so be careful.)

Re: SQLite is all you need for durable workflows

#227

Post by kukkeliskuu » Sat, May 30, 2026, 3:22 AM UTC

faangguyindia wrote: post #214

I've replaced all of these with Go + SQLite: 1. Intercom 2. Zendesk 3. Email marketing 4. Kanban 5. Todo 6. Our billing stack 7. Our issue tracker 8. Our forum 9. Uptime monitor 10. PagerDuty (clone) I have dozens of products I sell, so I thought: why not build everything ourselves? All of these run on the same server and use very little memory. I replaced all the SaaS tools we used with these. I also moved to dedica…

I am doing approximately the same for some of these as well, plus CRM. For me, CRM is just a list of contacts with a state and some other fields, plus record who was contacted and when. But I can directly pull information that is relevant to me. Integrating to an external CRM is more work and arguably more risky than rolling your own.

Re: SQLite is all you need for durable workflows

#228

Post by Rohansi » Sat, May 30, 2026, 3:45 AM UTC

apatheticonion wrote: post #224

levkk wrote: post #22

I don't understand this obsession with SQLite for real, production apps. SQLite is an embedded database, completely unsuitable for managing concurrency. This is what database _servers_ are for, e.g., Postgres, MySQL, etc. Their entire job is to allow you to modify data from multiple processes, on different machines, at the same time. This is a foundational principle of computer science. It seems to me that the "SQLit…

For me, I have a use case that needs to support a few thousand users, probably a few hundred concurrently. The combination of SQLite (libsql, a concurrent implementation of sqlite) and Rust means I can do so from a $2/m VPS and a single server instance. Backups are done via a cron job that uploads to S3. Does it pass the "Netflix scale" test? No But it doesn't need to. I'm not profiting from the service and SQLite of…

> The combination of SQLite (libsql, a concurrent implementation of sqlite) and Rust means I can do so from a $2/m VPS and a single server instance.

You can probably do it with regular SQLite, too. Being limited to a single writer isn't as devastating as it sounds when they get processed very quickly. Probably don't need Rust either but it'll be more efficient than the usual choices.

(Also, it looks like libsql is the same as SQLite? Only Turso has concurrent writes)

Re: SQLite is all you need for durable workflows

#229

Post by tucnak » Sat, May 30, 2026, 3:59 AM UTC

jrockway wrote: post #226

levkk wrote: post #22

I don't understand this obsession with SQLite for real, production apps. SQLite is an embedded database, completely unsuitable for managing concurrency. This is what database _servers_ are for, e.g., Postgres, MySQL, etc. Their entire job is to allow you to modify data from multiple processes, on different machines, at the same time. This is a foundational principle of computer science. It seems to me that the "SQLit…

There is something appealing about "it's just a file" (it really isn't; it has locks and a WAL), but I agree with you. I think people are afraid to read the documentation for postgres. You can start it up in milliseconds. Fast enough and light enough to run one copy for every test case in your test suite, or whatever you're using it for. (mkdir /tmp/whatever; initdb -D /tmp/whatever --no-instructions -A reject -c lis…

> I think people are afraid to read the documentation for postgres.

Postgres may introduce a single-file embedded filesystem because what the hell, but the irony is all these guys won't even notice it. The same people that say Postgres backups are too hard.

Re: SQLite is all you need for durable workflows

#230

Post by Rohansi » Sat, May 30, 2026, 4:06 AM UTC

petcat wrote: post #69

peterspath wrote: post #56

Earlier quoted context omitted.

That’s why there are billions of SQLite databases right? SQLite is likely used more than all other database engines combined. Billions and billions of copies of SQLite exist in the wild. SQLite is found in: Every Android device Every iPhone and iOS device Every Mac Every Windows 10/11 installation Every Firefox, Chrome, and Safari web browser Every instance of Skype Every instance of iTunes Every Dropbox client Every…

sqlite is great for the contacts app on your phone, but that's it. Hipp even said that it is not a replacement for a real multi-user, concurrent RDMS. Its primary competitor is "fsync".

SQLite is able to handle tens of thousands of write transactions per second on modern hardware. That is probably similar to or more than your real, multi-user, concurrent RDBMS.