Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite is all you need for durable workflows

obeli.sk

Post reply on HN
Print view View original

201–210 of 413 posts

Re: SQLite is all you need for durable workflows

#201

Post by bdangubic »

not_kurt_godel wrote: post #193
tasuki wrote: post #112

Earlier quoted context omitted.

But now you have another process to babysit. How do you keep it healthy? And you have to ensure the client-server communication won't break. For me the main benefit of sqlite is that it's a library rather than an app.

> But now you have another process to babysit. How do you keep it healthy? I've been assured by many HN users that running apps/sites on a single VPS requires near-zero maintenance or monitoring to achieve acceptable uptime 24/7/365 for years on end, sooooo...just pretend it will never fail like your main server process?

24/7/365 is needed (or achieved) just about never. our big tech is proving 90% will soon be utopia as well. being down has always been fine for 99.999975% of all projects on the planet.

Re: SQLite is all you need for durable workflows

#202

Post by psanford »

I wrote a library[0] to let you concurrently update a sqlite db in s3 safely. It uses the little known sqlite sessions extension plus s3 compare-and-swap on a small metadata file to make this work reasonably efficiently and safely. I have been enjoying it for a bunch of small projects where I want a lambda function to have a db for state but I don't want to pay for a full database instance.

[0]: https://github.com/psanford/s3db

Re: SQLite is all you need for durable workflows

#203

Post by pseudalopex »

formerly_proven wrote: post #144
Thaxll wrote: post #118

I started using SQLite for a home project after years of reading about it, I was shocked at the poor type system coming from Postgres. It is really inferior, not sure why it gets so much praise. https://sqlite.org/datatype3.html https://www.postgresql.org/docs/current/datatype.html Working with date/time feels like using a 30years old database, nothing is enforced at insert. Really someone needs to explain why so man…

This is the fault/price of backwards compatibility. Most users of SQLite should just fire off a few pragmas on each connection: PRAGMA journal_mode = WAL PRAGMA foreign_keys = ON # Something non-null PRAGMA busy_timeout = 1000 # This is fine for most applications, but see the manual PRAGMA synchronous = NORMAL # If you use it as a file format PRAGMA trusted_schema = OFF You might need additional options, depending on…

The price of compatibility could be a pragma.

Re: SQLite is all you need for durable workflows

#204

Post by andix »

kellogah wrote: post #189
abtinf wrote: post #58

Earlier quoted context omitted.

There are many cases where SQLite + concurrent front end (like a go net/http server) can handle all the load that a service might ever conceivably have to handle, especially if allowed to scale up hardware over time. You can trivially scale up SQLite to, what, hundreds of thousands of tps? The only thing you really give up is HA/failover and DR. But there are solutions to deal with those. And single-server systems ar…

I was thinking of using SQLite on top of k3s/Longhorn to replicate it. Anyone do something similar? Folks mention light steam and aws but Jeff Bezos’s biceps are too much for me to handle.

A longhorn volume can only be attached to one node at at time. It can share it with other nodes over nfs. I don't think this is going to scale well.

Just use Postgres with ro replicas.

Re: SQLite is all you need for durable workflows

#205

Post by horsawlarway »

kellogah wrote: post #189
abtinf wrote: post #58

Earlier quoted context omitted.

There are many cases where SQLite + concurrent front end (like a go net/http server) can handle all the load that a service might ever conceivably have to handle, especially if allowed to scale up hardware over time. You can trivially scale up SQLite to, what, hundreds of thousands of tps? The only thing you really give up is HA/failover and DR. But there are solutions to deal with those. And single-server systems ar…

I was thinking of using SQLite on top of k3s/Longhorn to replicate it. Anyone do something similar? Folks mention light steam and aws but Jeff Bezos’s biceps are too much for me to handle.

I'll echo the other response.

I've had pretty terrible experiences with SQLite and Longhorn/NFS.

It's just not the right database for pretty much ANY network based filesystem, where the locking primatives aren't as robust, and you might get two processes trying to hit it at the same time.

Frankly - they say this themselves: https://sqlite.org/howtocorrupt.html

As someone who runs a fairly big personal cluster backed by a mix of giant NFS storage for media, and relatively large longhorn SSD drives for configs/temp data...

I avoid sqlite backing like the plague. It will get corrupted. Period. It's not the db for this use-case, and I'll take postgres/maria/mysql/mongo/ANYTHING else over it.

If you do it - back it up ALL THE TIME, because it's going to get corrupted.

Re: SQLite is all you need for durable workflows

#206

Post by [deleted] »

tempest_ wrote: post #186
kenforthewin wrote: post #184

Earlier quoted context omitted.

Could you expand on the "substantial ops burden"? Let's say you're using a managed Postgres instance as the underlying data store, how substantial is the ops burden in that case? I understand that temporal is actually a set of 4 or so microservices on top of a data store, but if you're already running a distributed system backed by k8s or something like that, it doesn't seem like it adds significant incremental ops o…

As a dev I would tell you its an ops burden. My devops coworker just shrugs, pumps out some yaml and helm and away it goes. It really depends on your experience and tolerance for a lot of things. Usually maintenance burden doesent start to make itself known till you get off the happy path or something breaks. Sometimes it can be a long while before that happens, sometimes it happens right away.

[deleted]

Re: SQLite is all you need for durable workflows

#207

Post by fcarraldo »

bitexploder wrote: post #7

I started setting up my workflows using Temporal. It deploys as relatively light weight local app. For an isolated local installation it uses SQLite. It makes the process of dealing with API retries and organizing workflows and tasks really simple. I recommend giving it a try. It is, philosophically, exactly what this article is suggesting, but it adds an incredibly rich and flexible interface for agents to work with…

It sounds like you’re running this mostly on a single machine? Temporal gets much more complex with scale. Cassandra isn’t fun to manage. Ringpop and TChannel are hard to debug when things go wrong. The SQL backend support doesn’t support horizontally scaled replicas (just single instance) due to consistency requirements. Depending on how your code is written, modifying code baked into workflows becomes complex, as anything that modifies the history event ordering breaks determinism in already-deployed workers.

We use it heavily and everyone who started on it doing simple scripting/automation all love it, everyone who built real production systems on top of it all hate it. Possibly operator error, but my experience hasn’t matched the rosy picture painted in these comments.

Re: SQLite is all you need for durable workflows

#208

Post by not_kurt_godel »

bdangubic wrote: post #201
not_kurt_godel wrote: post #193

Earlier quoted context omitted.

> But now you have another process to babysit. How do you keep it healthy? I've been assured by many HN users that running apps/sites on a single VPS requires near-zero maintenance or monitoring to achieve acceptable uptime 24/7/365 for years on end, sooooo...just pretend it will never fail like your main server process?

24/7/365 is needed (or achieved) just about never. our big tech is proving 90% will soon be utopia as well. being down has always been fine for 99.999975% of all projects on the planet.

Ok, now tell me the stat by percentage of overall market revenue rather than project count

Re: SQLite is all you need for durable workflows

#209

Post by platz »

bitexploder wrote: post #164
svara wrote: post #113

Earlier quoted context omitted.

Word on HN is that you're either paying more money than you expected for temporal's managed solution or taking on substantial ops burden ultimately running their very heavy system yourself. I wouldn't know, I've not done either, but I'd like to learn more from your or other's experience.

I told an agent to set it up for me for some local stuff. It is written in Go. It has a painless path to run on a local SQLite DB. My agents use it to organize and coordinate workflows. It handles retries and long horizon tasks fine. As far as I can tell for the core workflows and tasks pieces it’s great. MIT license. Like anything it isn’t free to manage but it offers a lot in return. High reliability systems are ha…

whats an example of things that you have your agents do that use workflows and sqlite db

Re: SQLite is all you need for durable workflows

#210

Post by not_kurt_godel »

ai_fry_ur_brain wrote: post #199
not_kurt_godel wrote: post #193

Earlier quoted context omitted.

> But now you have another process to babysit. How do you keep it healthy? I've been assured by many HN users that running apps/sites on a single VPS requires near-zero maintenance or monitoring to achieve acceptable uptime 24/7/365 for years on end, sooooo...just pretend it will never fail like your main server process?

Ive been assured by many HN users that you must have 24/7/365 uptime for everything in case one of your 10 bi-monthly users decides to log on.

Call me old-fashioned and quaint, but I don't like to build software that doesn't work all the time if I can help it, whether it's for 10 users or 10 million.
Post reply on HN