Live data from Hacker News

SQLite is all you need for durable workflows

obeli.sk

111–120 of 413 posts

Re: SQLite is all you need for durable workflows

#111

I started setting up my workflows using Temporal. It deploys as relatively light weight local app. For an isolated local installation it uses SQLite. It makes the process of dealing with API retries and organizing workflows and tasks really simple. I recommend giving it a try. It is, philosophically, exactly what this article is suggesting, but it adds an incredibly rich and flexible interface for agents to work with…

This reads like an advertisement for Temporal :)

Low key amazing tech, kinda like clickhouse - nobody is bragging it’s running their business

Re: SQLite is all you need for durable workflows

#112

Earlier quoted context omitted.

Well if you run a tiny single-threaded app then SQLite is a nice simplification over spinning up a separate machine for Postgres.

I use postgres for very simple apps. I have a Dockerfile I use in my boilerplate repo. It takes a single make cmd for me to build, start and run migrations. Its as simple as using sqlite.

But now you have another process to babysit. How do you keep it healthy? And you have to ensure the client-server communication won't break.

For me the main benefit of sqlite is that it's a library rather than an app.

Re: SQLite is all you need for durable workflows

#113

I started setting up my workflows using Temporal. It deploys as relatively light weight local app. For an isolated local installation it uses SQLite. It makes the process of dealing with API retries and organizing workflows and tasks really simple. I recommend giving it a try. It is, philosophically, exactly what this article is suggesting, but it adds an incredibly rich and flexible interface for agents to work with…

Word on HN is that you're either paying more money than you expected for temporal's managed solution or taking on substantial ops burden ultimately running their very heavy system yourself.

I wouldn't know, I've not done either, but I'd like to learn more from your or other's experience.

Re: SQLite is all you need for durable workflows

#114

Related: Building durable workflows on Postgres https://news.ycombinator.com/item?id=48313530

Surfacing for the thread:

Armin Ronacher's "Absurd Workflows: Durable Execution With Just Postgres" https://lucumr.pocoo.org/2025/11/3/absurd-workflows/

Re: SQLite is all you need for durable workflows

#116
> Postgres ... is the right choice when you need higher availability, broader shared scalability, or other deployment properties that are better served by a network database. It is also the better fit when asynchronous replication to object storage is not the durability model you want... Many workflow systems do not need that on day one and should not start with more infrastructure than their state actually demands.

------

I see this kind of YAGNI thinking a lot, but in my view, it must be balanced against the effort you'd put into resolving any edge cases and adapting current architecture to your use case.

Imagine you deploy Sqlite, and thought it fine by itself, you keep running into some unforeseen challenges with the use to which you are putting. YOu'd need to sink valuable time and effort into addressing those. Then, when you have outgrown it, you'd beed to spend additional valuable times dping the same with Postgres.

This is why, when it comes to Architecture, I increasingly find my myself over-enigneering a bit. Assuming there is a good chance you might need to upgrade your architecture in the not too distant future, that approach is actually kind of very efficient. I find that I am able to uncover a lot of potential gotchas, which feeds back into the what the simplified current architecture should be, and helps me understand the roadmap I'm facing very well. I also avoid wasting too much time going too deep in directions that make sense now, but need a lot of plumbing to get right, when I can see that I'd likely have to throw it all out in a few years. Going from A -> B -C -> D, where each step is the optimal good-enough-for-now architecture but which requires a lot of work to stabilize and iron out the kinks of, is much less efficient than exploring D well enough to know whether you should build A, B, or C now.

Basically, some over-engineering, if done right, is not wasted. It cuts right to the heart of what you are dealing with, efficiently, and allows you to make (maybe) simpler but informed choices now as to how best to allocate your development resources now.

Re: SQLite is all you need for durable workflows

#117
post #84
post #53

Earlier quoted context omitted.

I had very good results giving 1 SQL DB per go routine, so the accesses were serialized up front, on a very high volume (130K requests/second) service. Exact transactionality was not a product goal, and the SQLite was just to backup the in memory state. If we lost a little due to abend or something, that was ok (although for normal maintenance it caught SIGTERM and stopped the listen and then waited for in flight cal…

Woof. That sounds very complicated. If you need that kind of write concurrency, use an unlogged table in postgres [0]. Then you don't have to invent a whole sharded thing yourself. [0] https://www.postgresql.org/docs/current/sql-createtable.html...

There are so many unfortunate footguns with unlogged tables, that I'd argue that the goroutine route is preferable.

Re: SQLite is all you need for durable workflows

#118
I started using SQLite for a home project after years of reading about it, I was shocked at the poor type system coming from Postgres. It is really inferior, not sure why it gets so much praise.

https://sqlite.org/datatype3.html

https://www.postgresql.org/docs/current/datatype.html

Working with date/time feels like using a 30years old database, nothing is enforced at insert. Really someone needs to explain why so many people like it.

Re: SQLite is all you need for durable workflows

#119
post #22

I don't understand this obsession with SQLite for real, production apps. SQLite is an embedded database, completely unsuitable for managing concurrency. This is what database _servers_ are for, e.g., Postgres, MySQL, etc. Their entire job is to allow you to modify data from multiple processes, on different machines, at the same time. This is a foundational principle of computer science. It seems to me that the "SQLit…

Scale to zero is very useful.

Re: SQLite is all you need for durable workflows

#120
if you have an application that needs to maintain state in a non-critical section or if you discover that using SQL is actually a good idea for some tasks (even in critical sections), SQLite is not only a good choice but it will save you a lot of time coming up with a brittle custom solution.

maintain an in-memory SQLite db and work it with SQL commands, and if you also want to preserve state across application restarts you can routinely save to disk or load from it: https://www.sqlite.org/backup.html#example_1_loading_and_sav...>

this also happens to be the most convenient file-format (aka. application-format) I ever worked with.

Post reply on HN