[flagged]
SQLite is all you need for durable workflows
271–280 of 413 posts
Re: SQLite is all you need for durable workflows
#272I started setting up my workflows using Temporal. It deploys as relatively light weight local app. For an isolated local installation it uses SQLite. It makes the process of dealing with API retries and organizing workflows and tasks really simple. I recommend giving it a try. It is, philosophically, exactly what this article is suggesting, but it adds an incredibly rich and flexible interface for agents to work with…
It sounds like you’re running this mostly on a single machine? Temporal gets much more complex with scale. Cassandra isn’t fun to manage. Ringpop and TChannel are hard to debug when things go wrong. The SQL backend support doesn’t support horizontally scaled replicas (just single instance) due to consistency requirements. Depending on how your code is written, modifying code baked into workflows becomes complex, as a…
> Depending on how your code is written, modifying code baked into workflows becomes complex, as anything that modifies the history event ordering breaks determinism in already-deployed workers.
I see this as Temporal surfacing inherent complexity of the domain in a way that forces the developer to consider it, rather than introducing extra complexity.
If it didn't make workflow determinism a strict requirement, the requirement would still exist - it would just hurt much worse in production when it's broken.
See also: Rust borrowing
Re: SQLite is all you need for durable workflows
#273Quote 1: "DBOS recently argued that Postgres is all you need for durable execution...SQLite is all you need."
Quote 2: "SQLite State backed up to S3". Yeah buddy, if you think S3 isn't Postgres I'll eat my foot.
Re: SQLite is all you need for durable workflows
#274Earlier quoted context omitted.
Thing is SQLite scales better than both those network databases [1] if you're prepared to stick with one big machine (+ a standby). This is even more obvious when you start doing transactions processing an row locks across the network limit you to 1-3k TPS that you cannot scale out of (Pareto distribution is merciless). [1] - https://andersmurphy.com/2025/12/02/100000-tps-over-a-billio...
Seeing as I can get about 200K TPS from a networked DB in my environment, I have to question your setup here. In the real world we are looking at things like RPO (recovery point objective) and RTO (recovery time objective). You need to consider HA and DR. It’s in these areas where SQLite does not scale. That’s why I struggle to see the fit for SQLite in any sort of multi-user server environment. If you need the data…
In this demo each T in TPS is two updates over a billion rows and most importantly skewing high on row lock contention. On a 5 year old macbook, using a dynamic language. Isolation level serializable and synchronous full (so max durability).
You can definitely go faster over less data doing single inserts on a better stack, with weaker guarantees.
RPO litestream even in it's default settings gives you point in time streaming backups to the second, which is considerably better than what RDS five minutes. So the funny thing is the durability guarantees are worse with the "bigger DBs".
RTO again you can have a standby that's warm with a copy of the data through litestream. Regional sharding also becomes trivial.
It's a solid set up for a lot of products/apps. Postgres is still fine if you want things like roles and permissions etc. Or if you don't have experience getting the most out of sqlite.
Re: SQLite is all you need for durable workflows
#275Earlier quoted context omitted.
Maybe that too is a native question, but there's a large scale between single user and 6000 tweets per second - most of our apps will never reach anything approaching even one save a second. So where to draw the line? I do far have gone the sqlite route for my hobby apps as it's so easy to handle and doesn't require setting up two docker containers for a single app. Am I drawing myself in a corner in case my apps eve…
Excellent question, and I spent so many years asking myself it, this over and over. You asking it made me realize I just...don't anymore. So allow me to blather a bit / free associate because I won't be sure why myself until I've written it out. TL;DR: whatever works for you is the right decision. (which isn't helpful, I heard this so many times and as the recipient, I thought "That's nice. Now how do I choose what w…
Thank you for sharing a detailed anecdote from production; there's not many of those around here.
Re: SQLite is all you need for durable workflows
#276Earlier quoted context omitted.
SQLite is good for read-concurrency, not great for write-concurrency.
SQLite requires writes run sequentially. Most SQLite write operations take single digit milliseconds or even microseconds. If your writes are inexpensive (inserting or updating single small rows) you'll probably never even notice the queue.
Exactly, people confuse "doesn't scale" with "is a bottleneck". There's many applications whereby hitting the limits of SQLite is either a physical impossibility, or implies that the application has achieved success such that replacing SQLite is the least of anyone's problems.
I visited a piano store once that was running everything off MS Access. If only they had switched to HA technologies, they would be able to sell millions of pianos a day!
Re: SQLite is all you need for durable workflows
#277I've replaced all of these with Go + SQLite: 1. Intercom 2. Zendesk 3. Email marketing 4. Kanban 5. Todo 6. Our billing stack 7. Our issue tracker 8. Our forum 9. Uptime monitor 10. PagerDuty (clone) I have dozens of products I sell, so I thought: why not build everything ourselves? All of these run on the same server and use very little memory. I replaced all the SaaS tools we used with these. I also moved to dedica…
This is the way, with "bus factor" as the one downside.
What happens when you get hit by a bus or someone with higher annual revenue tracks you down because of this comment and hires you away from this custom software stack with a bigger slice of the profits?
It sounds like there's also a bus factor for your server but I'm sure you're aware of that though it sounds like your clients aren't.
Re: SQLite is all you need for durable workflows
#278Next one: Berkeley DB is all you need for durable workflows /s
Re: SQLite is all you need for durable workflows
#279I don't understand this obsession with SQLite for real, production apps. SQLite is an embedded database, completely unsuitable for managing concurrency. This is what database _servers_ are for, e.g., Postgres, MySQL, etc. Their entire job is to allow you to modify data from multiple processes, on different machines, at the same time. This is a foundational principle of computer science. It seems to me that the "SQLit…
> SQLite for everything
is just wrong, and I don't think that the SQLite fans are that crowd. Taking a database server for everything is probably possible, but often unnecessary. With experience, one can properly judge when SQLite is sufficient and when it is not.
So arguing that the SQLite crowd is inexperienced feels weird, because inexperienced people have a much harder time judging when to use what and can just use the database server all the time (even when it is overkill).
Re: SQLite is all you need for durable workflows
#280Earlier quoted context omitted.
And I don't understand the obsession with server-based databases for single apps. Especially in containerised setups, every "app" gets its own database anyways, and if the app is further broken down into services, they usually communicate between each other and not with a shared database. So in those cases, what do you gain by pulling the database out of the "process" and onto the other end of a socket? In most cases…
“Especially in containerised setups, every "app" gets its own database anyways, and if the app is further broken down into services, they usually communicate between each other and not with a shared database. “ You seem to be talking about a vastly different use case. Containerized apps having their own database? What? Aren’t these types of containers stateless? I always very much try to keep state out of app contain…
If an app needs a database, it gets a database server container, instead of getting a user and database on a shared database server as things used to be done. Every little django app has its own postgres container. Every wordpress site gets its own mysql container. That is the modern way.
Those database containers get a PVC/volume/mount for their data dirs. The only thing ever connecting to them is their "owner" application container. So at that point, why not drop the postgres container and PVC mount a sqlite directory in the app container? The result is the same.