Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite is all you need for durable workflows

obeli.sk

Post reply on HN
Print view View original

311–320 of 413 posts

Re: SQLite is all you need for durable workflows

#311

Post by jappgar »

franga2000 wrote: post #280
stingraycharles wrote: post #261

Earlier quoted context omitted.

“Especially in containerised setups, every "app" gets its own database anyways, and if the app is further broken down into services, they usually communicate between each other and not with a shared database. “ You seem to be talking about a vastly different use case. Containerized apps having their own database? What? Aren’t these types of containers stateless? I always very much try to keep state out of app contain…

If an app needs a database, it gets a database server container, instead of getting a user and database on a shared database server as things used to be done. Every little django app has its own postgres container. Every wordpress site gets its own mysql container. That is the modern way. Those database containers get a PVC/volume/mount for their data dirs. The only thing ever connecting to them is their "owner" appl…

And when you need to scale to thousands of instances of your microservice?

Re: SQLite is all you need for durable workflows

#312

Post by jappgar »

ummonk wrote: post #234
abtinf wrote: post #58

Earlier quoted context omitted.

There are many cases where SQLite + concurrent front end (like a go net/http server) can handle all the load that a service might ever conceivably have to handle, especially if allowed to scale up hardware over time. You can trivially scale up SQLite to, what, hundreds of thousands of tps? The only thing you really give up is HA/failover and DR. But there are solutions to deal with those. And single-server systems ar…

Why go through the trouble of shoehorning SQLite into a cloud database by getting solutions for HA/failover and DR, when you can just use Postgres off the shelf?

So you can post about it on HN, obviously

Re: SQLite is all you need for durable workflows

#313

Post by [deleted] »

mxey wrote: post #294
levkk wrote: post #22

I don't understand this obsession with SQLite for real, production apps. SQLite is an embedded database, completely unsuitable for managing concurrency. This is what database _servers_ are for, e.g., Postgres, MySQL, etc. Their entire job is to allow you to modify data from multiple processes, on different machines, at the same time. This is a foundational principle of computer science. It seems to me that the "SQLit…

I think the SQLite website itself says it best: > SQLite does not compete with client/server databases. SQLite competes with fopen().

[deleted]

Re: SQLite is all you need for durable workflows

#314

Post by stingraycharles »

jappgar wrote: post #311
franga2000 wrote: post #280

Earlier quoted context omitted.

If an app needs a database, it gets a database server container, instead of getting a user and database on a shared database server as things used to be done. Every little django app has its own postgres container. Every wordpress site gets its own mysql container. That is the modern way. Those database containers get a PVC/volume/mount for their data dirs. The only thing ever connecting to them is their "owner" appl…

And when you need to scale to thousands of instances of your microservice?

Yeah this is the part I don’t get. It seems like people are talking about 1 distinct app = 1 container and this is the new normal? We’re back to managing cows instead of cattle again?

Re: SQLite is all you need for durable workflows

#315

Post by tauwauwau »

trueno wrote: post #248
bitexploder wrote: post #238

Earlier quoted context omitted.

Autonomous C to Rust. Automated penetration testing and vuln validation.

you just made me realize how much i wished people stopped talking in abstractions and just stated what they were doing. i hadnt realized how often i saw things like "workflows" and just kinda had my eyes glaze over. none of it ever really clicks until i see the true descriptor of whats going on. ive been over here using claude relatively simply as of recent, just claude code and i might enter plan mode to do some big…

Check out Matt Pocock's coding workflow. His approach is repeatable, consistent and is backed backed by actual theories in large software development.

Re: SQLite is all you need for durable workflows

#316

Post by UnfitFootprint »

inglor wrote: post #266
fcarraldo wrote: post #207

Earlier quoted context omitted.

It sounds like you’re running this mostly on a single machine? Temporal gets much more complex with scale. Cassandra isn’t fun to manage. Ringpop and TChannel are hard to debug when things go wrong. The SQL backend support doesn’t support horizontally scaled replicas (just single instance) due to consistency requirements. Depending on how your code is written, modifying code baked into workflows becomes complex, as a…

In the last two years, we built (with a team of 15, now 100) a billion dollar business on top of Temporal that performs business critical applications for fortune 500 companies. We couldn't be happier with temporal. Determinism sucks, you do have to work hard and make everything idempotent in activities like we would for durable software anyway. The language we used was incorrect (Go) and has a lot of boilerplate com…

Could you share a bit more about your learnings on go + temporal? That combo was next in line for us to migrate _to_

Re: SQLite is all you need for durable workflows

#318

Post by nickpeterson »

IshKebab wrote: post #137
Thaxll wrote: post #118

I started using SQLite for a home project after years of reading about it, I was shocked at the poor type system coming from Postgres. It is really inferior, not sure why it gets so much praise. https://sqlite.org/datatype3.html https://www.postgresql.org/docs/current/datatype.html Working with date/time feels like using a 30years old database, nothing is enforced at insert. Really someone needs to explain why so man…

It gets praise because of stuff other than the type system. I agree it is disappointing, especially before strict tables. You should check out DuckDB which is basically SQLite but with proper types. Although it is also OLAP (struct of arrays) rather than OLTP (array of structs) which may have worse performance for typical SQLite loads. In practice I doubt it matters if you have an application where you're considering…

A lot of oltp databases have modeling conventions for making read only reporting tables. Do tabular dbs have an inverse for transaction heavy data, that later gets batched to a read optimized structure? I kind of think most databases (even oltp workloads) really are read dominated. I feel like DuckDB is really close to working as the ‘main db’ for such systems, but my lack of knowledge for how to handle quick mutations bothers me. It feels like some form of temporal data modeling would solve it but I don’t know.

Re: SQLite is all you need for durable workflows

#319

Post by emehex »

therealdrag0 wrote: post #176
emehex wrote: post #138

Earlier quoted context omitted.

I think you'd be surprised to learn how many real production apps are actually running on top of SQLite (by way of Cloudflare D1).

Many DB servers are built upon embedded DB primitives (like RocksDB), that doesn’t mean the primitives are sufficient on their own.

I'm not sure what this has to do with my comment? D1 is pretty much sufficient on it's own...
Post reply on HN