Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite is all you need for durable workflows

obeli.sk

Post reply on HN
Print view View original

161–170 of 413 posts

Re: SQLite is all you need for durable workflows

#161

Post by therealdrag0 »

MattJ100 wrote: post #143
tasuki wrote: post #107

Earlier quoted context omitted.

> SQLite can work well for the parts of your system where there is naturally strong partitioning. Or the parts of your system that don't have big data and no need for massively concurrent writes. And that's the vast majority of systems!

You can do big data in SQLite. Concurrent writes, sure, I'd recommend something else. If you think the majority of systems require massively concurrent writes, I think you need to look a bit harder. SQLite is, after all, the most widely deployed database system, ever.

It’s widely deployed as a local DB for local apps like phones, desktops, and web browsers. But it’s not the most used for distributed, concurrent web apps, which db servers were designed for. Maybe people are talking past each other, but that’s the debate I see.

Re: SQLite is all you need for durable workflows

#162

Post by simonw »

nodesocket wrote: post #135

The biggest annoyance about SQLite for me is no ability to: ALTER TABLE users MODIFY COLUMN… ALTER TABLE users ALTER COLUMN… ALTER TABLE users ADD CONSTRAINT… You have to create a new temporary table with correct schema, copy data into this new table, drop the old table, and then rename the temporary table.

They've been improving that recently:

2026-04-09 (3.53.0) - "Enhance ALTER TABLE to permit adding and removing NOT NULL and CHECK constraints"

I use my own sqlite-utils CLI/Python library to work around these limitations: https://sqlite-utils.datasette.io/en/stable/python-api.html#...

Re: SQLite is all you need for durable workflows

#163

Post by gchamonlive »

wolttam wrote: post #78
gchamonlive wrote: post #66

Earlier quoted context omitted.

Are logs all you need for durable workflows? I'm confused here. How'd persist and query nested or related data over logs? By logs I assume you mean something like elasticsearch or meilisearch?

Pretty much every durable system has an intent log of some sort. The log provides durability, the database system just integrates that log into a more queryable format.

I swear it didn't occur to me that that mean WAL, makes much more sense now LOL

Re: SQLite is all you need for durable workflows

#164

Post by bitexploder »

svara wrote: post #113
bitexploder wrote: post #7

I started setting up my workflows using Temporal. It deploys as relatively light weight local app. For an isolated local installation it uses SQLite. It makes the process of dealing with API retries and organizing workflows and tasks really simple. I recommend giving it a try. It is, philosophically, exactly what this article is suggesting, but it adds an incredibly rich and flexible interface for agents to work with…

Word on HN is that you're either paying more money than you expected for temporal's managed solution or taking on substantial ops burden ultimately running their very heavy system yourself. I wouldn't know, I've not done either, but I'd like to learn more from your or other's experience.

I told an agent to set it up for me for some local stuff. It is written in Go. It has a painless path to run on a local SQLite DB. My agents use it to organize and coordinate workflows. It handles retries and long horizon tasks fine. As far as I can tell for the core workflows and tasks pieces it’s great. MIT license. Like anything it isn’t free to manage but it offers a lot in return. High reliability systems are hard. Temporal only solves some of it. It is far better than rolling it yourself.

I think a genuine problem right now is people are building agentic work flows and learning the hard way highly reliable agentic work flows are hard. Agents are unreliable. They are both not deterministic and not the backing APIs have pretty high error rates. Temporal has solved that pain for me and made it easy to diagnose problems.

I don’t have anything really large scale running. But big enough that it takes billions of tokens and high reliability to finish.

Re: SQLite is all you need for durable workflows

#165

Post by bitexploder »

embedding-shape wrote: post #106
fragmede wrote: post #41

Earlier quoted context omitted.

Honest answer is: whenever your markdown or json files get to be big enough that grep/jq takes long enough that you get bored waiting for it.

> get to be big enough that grep/jq takes long enough On a modern processor, that's about GBs of data typically, right?

Practically yes, but much earlier if agents are touching that data in my experience. Tens of GB even if you design well.

Re: SQLite is all you need for durable workflows

#166

Post by nitwit005 »

switchbak wrote: post #79
levkk wrote: post #22

I don't understand this obsession with SQLite for real, production apps. SQLite is an embedded database, completely unsuitable for managing concurrency. This is what database _servers_ are for, e.g., Postgres, MySQL, etc. Their entire job is to allow you to modify data from multiple processes, on different machines, at the same time. This is a foundational principle of computer science. It seems to me that the "SQLit…

I mean - I agree for the typical multi-user, SaaS webapp. But I don't think that's what these folks are proposing. If they are - yeesh, count me out. If on the other hand they're talking about single-user, software in the small - hell yeah. In fact, I'd also promote DuckDB in this regard (mostly for analytics) - with the power of a single machine these days, you can do a surprising amount and never have to worry abou…

The reason the parent post is complaining that it doesn't make sense, is because people have indeed pushed the idea of using SQLite as an alternative for web apps like that.

Re: SQLite is all you need for durable workflows

#168

Post by bitexploder »

peterson_lock wrote: post #38
bitexploder wrote: post #7

I started setting up my workflows using Temporal. It deploys as relatively light weight local app. For an isolated local installation it uses SQLite. It makes the process of dealing with API retries and organizing workflows and tasks really simple. I recommend giving it a try. It is, philosophically, exactly what this article is suggesting, but it adds an incredibly rich and flexible interface for agents to work with…

This reads like an advertisement for Temporal :)

Well, just my experience. I installed it, had my agents configure it and it immediately solved problems I had with very little friction. Dealing with long running, long horizon agentic tasks that need very high reliability so I don’t have to babysit. I vibed the first version, realized I was reinventing reliable distributed systems. Stopped vibing and started surveying for something that fit :)

Re: SQLite is all you need for durable workflows

#170

Post by nodesocket »

simonw wrote: post #162
nodesocket wrote: post #135

The biggest annoyance about SQLite for me is no ability to: ALTER TABLE users MODIFY COLUMN… ALTER TABLE users ALTER COLUMN… ALTER TABLE users ADD CONSTRAINT… You have to create a new temporary table with correct schema, copy data into this new table, drop the old table, and then rename the temporary table.

They've been improving that recently: 2026-04-09 (3.53.0) - "Enhance ALTER TABLE to permit adding and removing NOT NULL and CHECK constraints" I use my own sqlite-utils CLI/Python library to work around these limitations: https://sqlite-utils.datasette.io/en/stable/python-api.html#...

Ahh that's very nice. Unfortunately the default version of sqlite3 provided by Debian Trixie is 3.46.1.
Post reply on HN