Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite is all you need for durable workflows

obeli.sk

Post reply on HN
Print view View original

251–260 of 413 posts

Re: SQLite is all you need for durable workflows

#251

Post by sharts »

MattJ100 wrote: post #143
tasuki wrote: post #107

Earlier quoted context omitted.

> SQLite can work well for the parts of your system where there is naturally strong partitioning. Or the parts of your system that don't have big data and no need for massively concurrent writes. And that's the vast majority of systems!

You can do big data in SQLite. Concurrent writes, sure, I'd recommend something else. If you think the majority of systems require massively concurrent writes, I think you need to look a bit harder. SQLite is, after all, the most widely deployed database system, ever.

Internet Explorer 6 was the most widely deployed awesome piece of software. Those that hated it need to look a bit harder.

Re: SQLite is all you need for durable workflows

#256

Post by andersmurphy »

evdubs wrote: post #245
andersmurphy wrote: post #141

Earlier quoted context omitted.

Thing is SQLite scales better than both those network databases [1] if you're prepared to stick with one big machine (+ a standby). This is even more obvious when you start doing transactions processing an row locks across the network limit you to 1-3k TPS that you cannot scale out of (Pareto distribution is merciless). [1] - https://andersmurphy.com/2025/12/02/100000-tps-over-a-billio...

Wow, what an apples and aliens comparison. You add a bunch of transaction delays to your postgresql case because you can access a database over a network, but you use transaction batching for sqlite? Maybe just compare a local postgresql with/without batching to a local sqlite with/without batching to be much less misleading.

Because local postgres is a bad time unlesss it's the only thing running on the server. Even then sqlite will smoke postgres (even with unix sockets).

The point is to survive the Pareto row locking problem you need to move away from a network database (if you want to still have interactive transactions). The network part is the main point of a network database, once you drop that there's not much pointing sticking with the added complexity unless there's another feature you really need.

Re: SQLite is all you need for durable workflows

#257

Post by andersmurphy »

slashdave wrote: post #243
andersmurphy wrote: post #141

Earlier quoted context omitted.

Thing is SQLite scales better than both those network databases [1] if you're prepared to stick with one big machine (+ a standby). This is even more obvious when you start doing transactions processing an row locks across the network limit you to 1-3k TPS that you cannot scale out of (Pareto distribution is merciless). [1] - https://andersmurphy.com/2025/12/02/100000-tps-over-a-billio...

You know you can host a database like Postgres on the same machine, right?

Yes, it's still slower on the same machine, even with unix domain sockets.

It doesn't play nice with other things running with it in practice. JVM and postgres on the same box is a textbook bad time.

Re: SQLite is all you need for durable workflows

#258

Post by shevy-java »

Hmmm. SQLite is great, but I remember years ago, at a university cluster, I had to populate a SQL database via tons of INSERT statements from genomic/meta-genomic workflows. Postgresql was so much faster just at that particular action (inserting data) that it convinced me that SQLite may be useful for many, many applications, but for "big data"(sets), Postgresql is simply better.

Re: SQLite is all you need for durable workflows

#259

Post by sofixa »

chaps wrote: post #62
jawns wrote: post #9

Earlier quoted context omitted.

Could you give an example of a case where you'd use SQLite instead of jq or grep through Markdown?

The moment my JSON has any sort of depth and I need to write a parser for it and potentially account for unspecified behavior. JSON's nice when it's nice, but it's terrible when it's terrible. It's 100x easier to write SQL than writing jq and... dear god if I have to use grep -A or -B, I'm doing something wrong. Constraints are actually a good thing! The underlying database isn't the most important thing. Just use SQ…

> It's 100x easier to write SQL than writing jq and... dear god if I have to use grep -A or -B, I'm doing something wrong. Constraints are actually a good thing!

As an occasional consumer of JSON/CSV, that's why I really like DuckDB, it's just SQL for such file formats. And it manages to be super fast at it too.

Re: SQLite is all you need for durable workflows

#260

Post by burner_acc001 »

kumulo wrote: post #249
fcarraldo wrote: post #207

Earlier quoted context omitted.

It sounds like you’re running this mostly on a single machine? Temporal gets much more complex with scale. Cassandra isn’t fun to manage. Ringpop and TChannel are hard to debug when things go wrong. The SQL backend support doesn’t support horizontally scaled replicas (just single instance) due to consistency requirements. Depending on how your code is written, modifying code baked into workflows becomes complex, as a…

Temporal feels massive, I tried it for a small workflow embedded on my system, and worked fine, but when thinking on scaling it, it just didn't made any sense for my use case. I also have restate.dev on my reseearch list, which on paper should scale well and be definitely more lightweight and simple to setup, worth having a look.

give conductor a try. runs with sqlite or if you want to run it on a server mysql or postgres. I know people running with postgres with a decent scale and it works just fine.

https://docs.conductor-oss.org/

Post reply on HN