Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite is all you need for durable workflows

obeli.sk

Post reply on HN
Print view View original

331–340 of 413 posts

Re: SQLite is all you need for durable workflows

#331

Post by graerg »

jappgar wrote: post #311
franga2000 wrote: post #280

Earlier quoted context omitted.

If an app needs a database, it gets a database server container, instead of getting a user and database on a shared database server as things used to be done. Every little django app has its own postgres container. Every wordpress site gets its own mysql container. That is the modern way. Those database containers get a PVC/volume/mount for their data dirs. The only thing ever connecting to them is their "owner" appl…

And when you need to scale to thousands of instances of your microservice?

That's the whole thesis; YAGNI.

Re: SQLite is all you need for durable workflows

#332

Post by simon84 »

trueno wrote: post #248
bitexploder wrote: post #238

Earlier quoted context omitted.

Autonomous C to Rust. Automated penetration testing and vuln validation.

you just made me realize how much i wished people stopped talking in abstractions and just stated what they were doing. i hadnt realized how often i saw things like "workflows" and just kinda had my eyes glaze over. none of it ever really clicks until i see the true descriptor of whats going on. ive been over here using claude relatively simply as of recent, just claude code and i might enter plan mode to do some big…

About the same feeling here. I guess not everything is about global banking scale.

I've tried clever tricks to get AI produce unsupervised stuff and came back from it. The slop and loss of cognitive knowledge about what it did was uncomfortable to me... I cannot understand how you would hand off critical job to it.

Re: SQLite is all you need for durable workflows

#333

Post by jappgar »

stingraycharles wrote: post #314
jappgar wrote: post #311

Earlier quoted context omitted.

And when you need to scale to thousands of instances of your microservice?

Yeah this is the part I don’t get. It seems like people are talking about 1 distinct app = 1 container and this is the new normal? We’re back to managing cows instead of cattle again?

I just think a lot of people here haven't ever worked on large scale systems. They don't know what the don't know.

Re: SQLite is all you need for durable workflows

#334

Post by irons »

https443 wrote: post #321
golem14 wrote: post #36

Litestream releases 5.9 and newer have a bug that causes instances to sync an insane amount of data. a DB with <10K of data in it and practically no writes/reads causes something like 10GB of daily replication traffic. For my toy project that got needlessly expensive.

Is this bug logged?

Looks like https://github.com/benbjohnson/litestream/issues/1197. Still open as of now, with a potential cause noted in the latest comment.

Re: SQLite is all you need for durable workflows

#336

Post by dust42 »

peterson_lock wrote: post #38
bitexploder wrote: post #7

I started setting up my workflows using Temporal. It deploys as relatively light weight local app. For an isolated local installation it uses SQLite. It makes the process of dealing with API retries and organizing workflows and tasks really simple. I recommend giving it a try. It is, philosophically, exactly what this article is suggesting, but it adds an incredibly rich and flexible interface for agents to work with…

This reads like an advertisement for Temporal :)

I just spent the last two weeks digging into workflow state engines and temporal was one of the candidates. It is a VC backed fork of Cadence. The got 0.3B funding and whatever positive I read about them on the net I take with a big spoon of salt. Just my 2 cents.

Re: SQLite is all you need for durable workflows

#337

Post by lanstin »

jklmnopqrstuvw wrote: post #291
glzone1 wrote: post #246

Earlier quoted context omitted.

Running postgresql is an order of magnitude more complicated than sqlite. 130k tps even with unlogged is not always super easy especially if getting hit concurrently. Postgresql connection overhead alone can be pretty brutal if you are setting up and tearing down connections or have 1,000 writers etc. Postgresql generally requires good network connectivity. Folks doing sqlite distributed tend to have everything indep…

even with wal, postgresql can easily reach 130k tps in pipeline mode.

That was per container, with 16 containers per data center, so would be a lot of DBA tickets to get something that large; SQLite scaled with the horizontal scaling of the app; and we did have a flaky network - something like one in 100,000 tcp connections would fail. And occasionally the whole network would just go away for a number seconds. And the persistent container storage was managed by the same storage team that managed storage for the DB team, so base scalability and availability high.

Re: SQLite is all you need for durable workflows

#338

Post by DANmode »

ksd482 wrote: post #68
peterspath wrote: post #56

Earlier quoted context omitted.

That’s why there are billions of SQLite databases right? SQLite is likely used more than all other database engines combined. Billions and billions of copies of SQLite exist in the wild. SQLite is found in: Every Android device Every iPhone and iOS device Every Mac Every Windows 10/11 installation Every Firefox, Chrome, and Safari web browser Every instance of Skype Every instance of iTunes Every Dropbox client Every…

levkk is talking about concurrency. The list you gave doesn't explain high concurrency requirements for usage.

Why doesn’t each of your users have a SQLite database writing up to a main?

You can have as many as you want - and one is often plenty.

Re: SQLite is all you need for durable workflows

#339

Post by roryirvine »

stingraycharles wrote: post #304
utopiah wrote: post #265

The cycle of expertise : - what is X, I just do Y - wow I can see so many limits of Y, now I do X - I use X for literally everything - now that I properly understand the limits of Y but also the heavy constraints of X ... maybe Y is enough - I use Y for literally everything rinse & repeat. The thing is with actual usage and actual context one does learn and thus can get away with a lot more "basic" solution but it do…

Yup, when I look back at the silly stuff I did when I was somewhere in the middle (CQRS + event sourcing I’m looking at you), it’s interesting. It is a source of expertise, because you really learn a lot from it. But when you become old (43 over here), you really learn to appreciate “boring” solutions.

You also begin to recognise that the definition of "boring" changes over time, and - if you wait long enough - fashions begin to repeat themselves.

So, xBase was all you needed in the mid 80s. Then DBM was all you needed in the mid 2000s. Now, in the mid 2020s, we're told that it's SQLite that is all you need.

It was partly true then, and partly true now. But the full story's always been more complicated, so it's still worth considering a range of potential solutions rather than relying on simplistic rules of thumb or slogans.

(Wake me when nosql comes back into fashion, I'll be able to do a great "old man yells at clouds" routine about that one...)

Re: SQLite is all you need for durable workflows

#340

Post by velcrovan »

HighGoldstein wrote: post #308
Gud wrote: post #264

Earlier quoted context omitted.

It was not really “deployed” by a lot of people in the same sense. It was forced upon most of us(not me, I used BeOS then Debian then FreeBSD). I deployed phoenix.

The reason SQLite is the most deployed is that it's used by Android.

…and iOS, and Windows, and Mac OS, and Boeing, and Sony, and Firefox and Chrome and Safari…
Post reply on HN