SQLite is all you need for durable workflows • Hacker News Board

Re: SQLite is all you need for durable workflows

#381

Post by franga2000 » Sat, May 30, 2026, 8:40 PM UTC

cortesoft wrote: post #374

franga2000 wrote: post #239

Earlier quoted context omitted.

And I don't understand the obsession with server-based databases for single apps. Especially in containerised setups, every "app" gets its own database anyways, and if the app is further broken down into services, they usually communicate between each other and not with a shared database. So in those cases, what do you gain by pulling the database out of the "process" and onto the other end of a socket? In most cases…

How do you do server maintenance or handle hardware failure if your database is SQLite? You are going to have to take downtime, even in the best scenario.

1. Proxmox live migration or HA, Ceph storage

2. K8S DaemonSet, PVC backed by probably Ceph

3. Just..don't care? Do maintenance outside of working hours, fix issues quickly and explain things nicely to your customers. Not everything is google-scale. Most people can deal with some downtime.

And it's not like you won't have downtime in let's say a postgres-backed app. But now you have two "servers" to deal with.

Re: SQLite is all you need for durable workflows

#382

Post by hmaxdml » Sat, May 30, 2026, 8:51 PM UTC

bitexploder wrote: post #345

inglor wrote: post #266

Earlier quoted context omitted.

In the last two years, we built (with a team of 15, now 100) a billion dollar business on top of Temporal that performs business critical applications for fortune 500 companies. We couldn't be happier with temporal. Determinism sucks, you do have to work hard and make everything idempotent in activities like we would for durable software anyway. The language we used was incorrect (Go) and has a lot of boilerplate com…

That is the real truth people are voicing when they say Temporal is heavy. They are really saying: Durable, reliable, distributed workloads are hard and it takes effort to manage! And that is true. I know of no systems that make that genuinely easy. It is a hard discipline. Maybe Temporal makes that harder than it should be, but I have no experience there. There are no free lunches in this space. I have no idea how g…

Have you looked into DBOS? Same thesis: durable and reliable workflows are hard to manage -- it just doesn't have to be as hard as Temporal makes it be :)

Re: SQLite is all you need for durable workflows

#383

Post by hmaxdml » Sat, May 30, 2026, 8:54 PM UTC

cnlwsu wrote: post #325

fcarraldo wrote: post #207

Earlier quoted context omitted.

It sounds like you’re running this mostly on a single machine? Temporal gets much more complex with scale. Cassandra isn’t fun to manage. Ringpop and TChannel are hard to debug when things go wrong. The SQL backend support doesn’t support horizontally scaled replicas (just single instance) due to consistency requirements. Depending on how your code is written, modifying code baked into workflows becomes complex, as a…

We are a huge production setup where it’s absolutely critical successfully but we use temporal cloud. Hosting it yourself is what makes it miserable. https://temporal.io/resources/on-demand/netflix

That's why their entire business model -- like Astronomer's -- is geared toward cloud hosting. The architecture is so complex it takes a full time team to run it.

Re: SQLite is all you need for durable workflows

#384

Post by [deleted] » Sat, May 30, 2026, 9:02 PM UTC

inglor wrote: post #266

fcarraldo wrote: post #207

Earlier quoted context omitted.

It sounds like you’re running this mostly on a single machine? Temporal gets much more complex with scale. Cassandra isn’t fun to manage. Ringpop and TChannel are hard to debug when things go wrong. The SQL backend support doesn’t support horizontally scaled replicas (just single instance) due to consistency requirements. Depending on how your code is written, modifying code baked into workflows becomes complex, as a…

In the last two years, we built (with a team of 15, now 100) a billion dollar business on top of Temporal that performs business critical applications for fortune 500 companies. We couldn't be happier with temporal. Determinism sucks, you do have to work hard and make everything idempotent in activities like we would for durable software anyway. The language we used was incorrect (Go) and has a lot of boilerplate com…

[deleted]

Re: SQLite is all you need for durable workflows

#385

Post by mrkeen » Sat, May 30, 2026, 9:03 PM UTC

password4321 wrote: post #295

stephenlf wrote: post #64

Can’t wait to see the next iteration of this idea with “Logs are all you need for durable workflows.”

Pardon my ignorance trying to follow up on what is most likely sarcasm but is this not Kafka's claim to fame? I am joining a new project and need to know to what extent Kafka is still a part of the future for new big data projects. It doesn't seem like there are alternatives at the high end but instead the question is when other technologies (that are easier to manage, require less compute, etc.) max out.

> Pardon my ignorance trying to follow up on what is most likely sarcasm but is this not Kafka's claim to fame?

Yes

> I am joining a new project and need to know to what extent Kafka is still a part of the future for new big data projects.

It's not gonna win or die on merit.

If I were to sit here and propose that you mutate bank accounts in-place (which I'm more or less doing by analogy right now over on https://news.ycombinator.com/item?id=48339103), and you just need safe enough locking technology to do so, I'd be immediately and rightfully shouted down that no one does that and you write down the money transfers and derive the final balances from that.

Manage any other kind of state, however, by appending state-changes and rolling them into a derived state, and then you're chasing fads and doing resume-driven development. So I'm skeptical about the future of this way of doing things.

Re: SQLite is all you need for durable workflows

#386

Post by andersmurphy » Sat, May 30, 2026, 9:14 PM UTC

UltraSane wrote: post #376

andersmurphy wrote: post #372

Earlier quoted context omitted.

Seems fine at billions of rows in my experience.

what are your pragmas?

:cache_size 1562

:page_size 4096

:journal_mode "WAL"

:synchronous "NORMAL"

:temp_store "MEMORY"

:busy_timeout 5000

(Synchronous FULL in context where it matters)

But, it depends on the shape of your data, your indexes, how much of the data you care about is filling up a page. If your distribution is more even you sometimes need more cache than a more Pareto distributed data set etc. Things like not caching prepared statements costs you more (you should almost always be caching prepared statements per connection with sqlite).

You have to give things more thought at a billion sure. Partial indexes are your friend. You'll also want more cache to prevent thrashing etc.

- https://andersmurphy.com/2025/12/02/100000-tps-over-a-billio...

Re: SQLite is all you need for durable workflows

#387

Post by Culonavirus » Sat, May 30, 2026, 9:15 PM UTC

leowoo91 wrote: post #305

jph00 wrote: post #49

Earlier quoted context omitted.

You seem to have a rather limited understanding of what kinds of concurrency exist and how those needs are best met. Whether something is a server or not is not very relevant to this discussion. SQLite is an excellent production db for many real world workloads, as has been widely documented. It is very different to Postgres, so requires learning a whole new skill set. One way to think about it is that SQLite can wor…

What additional skill set do you need to "learn" for SQLite? Copying files around?

Yeah these are deeply unserious people.

Re: SQLite is all you need for durable workflows

#388

Post by sperandeo » Sat, May 30, 2026, 9:22 PM UTC

I concur. I keep waiting to hit the wall where I need Postgres and it just hasn't happened. PB ftw

Re: SQLite is all you need for durable workflows

#389

Post by tonymet » Sat, May 30, 2026, 9:53 PM UTC

faangguyindia wrote: post #218

tonymet wrote: post #216

Earlier quoted context omitted.

I rolled my own uptime but how do you vary region or do residential testing ?

I actually don't. I just have uptime service hosted outside of our main infra. It connects to my service called Siren, which alerts me on my phone with an alarm on full volume with SWAT cat intro. It's good enough for what we do, barely have any downtime. But it helped me figure out 6s downtime we would experience when our spot instances get knocked out, so it helped me increase health check frequency 6s downtime is…

ok makes sense, yeah I went with a similar approach. for a while I did a multi region thing but didn't see variance and it wasn't worth the trouble.

Re: SQLite is all you need for durable workflows

#390

Post by atreids » Sat, May 30, 2026, 9:59 PM UTC

gchamonlive wrote: post #302

gchamonlive wrote: post #159

Earlier quoted context omitted.

Is it common to use logs as a proxy for write-ahead logs?

Folks this is meant to be an honest question, not a snarky comment. I'm not a DBA, I'm DevOps/SRE and logs for me always meant execution logs. I'm just curious if between those involved in database domain logs is used to refer to WAL.

I think the original poster in this thread was joking. A fair number of databases use "logs" as a core mechanism for storing and sorting data. "Logs" in this context is not to be confused with stdout/stderr output that you may collect from a running program and forward somewhere like Cloudwatch/Elasticsearch etc. "Logs" in the context of databases here refers to the data structure; which can generally be defined as simply an append-only "file" (I put file in quotes because just because something is a log does not mean it is necessarily written to disk yet - that's why write-ahead logs exist). It's not just write-ahead logs.

Google "Log-structured merge trees" if you want an interesting read.