Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite is all you need for durable workflows

obeli.sk

Post reply on HN
Print view View original

181–190 of 413 posts

Re: SQLite is all you need for durable workflows

#181

Post by eterm »

s_ting765 wrote: post #96
eterm wrote: post #85

Earlier quoted context omitted.

Or you can run postgres on the same machine as the application, which lets you much more easily migrate if the time comes when you need to scale to multiple application servers. There's a world between "local file" and "network DB server", running a DB server locally has lots of benefits from being able to easily query from outside if needed to forcing you to consider concurrency without the latency overhead of a net…

This decision tree doesn't make much sense to me. Why you someone forego performance today in favor of adding a completely unnecessary network layer to every DB query in order to "satisfy" future imaginary "scaling concerns"?

Because you don't add a network layer by running a database locally.

Re: SQLite is all you need for durable workflows

#182

Post by eterm »

wat10000 wrote: post #153
eterm wrote: post #85

Earlier quoted context omitted.

Or you can run postgres on the same machine as the application, which lets you much more easily migrate if the time comes when you need to scale to multiple application servers. There's a world between "local file" and "network DB server", running a DB server locally has lots of benefits from being able to easily query from outside if needed to forcing you to consider concurrency without the latency overhead of a net…

Now you've added a substantial dependency, and annoying setup requirements. Good luck doing this for a native app on mobile or desktop.

Obviously SQLite is the best choice for a mobile or desktop app, that's not what's being discussed here.

Re: SQLite is all you need for durable workflows

#183

Post by friendly_deer »

stephenlf wrote: post #64

Can’t wait to see the next iteration of this idea with “Logs are all you need for durable workflows.”

In all seriousness, I’d take a “s3 is all you need for durable workflows” and use it in data processing applications that move data from s3 -> s3 with no other dependencies.

Re: SQLite is all you need for durable workflows

#184

Post by kenforthewin »

svara wrote: post #113
bitexploder wrote: post #7

I started setting up my workflows using Temporal. It deploys as relatively light weight local app. For an isolated local installation it uses SQLite. It makes the process of dealing with API retries and organizing workflows and tasks really simple. I recommend giving it a try. It is, philosophically, exactly what this article is suggesting, but it adds an incredibly rich and flexible interface for agents to work with…

Word on HN is that you're either paying more money than you expected for temporal's managed solution or taking on substantial ops burden ultimately running their very heavy system yourself. I wouldn't know, I've not done either, but I'd like to learn more from your or other's experience.

Could you expand on the "substantial ops burden"? Let's say you're using a managed Postgres instance as the underlying data store, how substantial is the ops burden in that case? I understand that temporal is actually a set of 4 or so microservices on top of a data store, but if you're already running a distributed system backed by k8s or something like that, it doesn't seem like it adds significant incremental ops on top of that. But I could be wrong.

Re: SQLite is all you need for durable workflows

#186

Post by tempest_ »

kenforthewin wrote: post #184
svara wrote: post #113

Earlier quoted context omitted.

Word on HN is that you're either paying more money than you expected for temporal's managed solution or taking on substantial ops burden ultimately running their very heavy system yourself. I wouldn't know, I've not done either, but I'd like to learn more from your or other's experience.

Could you expand on the "substantial ops burden"? Let's say you're using a managed Postgres instance as the underlying data store, how substantial is the ops burden in that case? I understand that temporal is actually a set of 4 or so microservices on top of a data store, but if you're already running a distributed system backed by k8s or something like that, it doesn't seem like it adds significant incremental ops o…

As a dev I would tell you its an ops burden.

My devops coworker just shrugs, pumps out some yaml and helm and away it goes.

It really depends on your experience and tolerance for a lot of things.

Usually maintenance burden doesent start to make itself known till you get off the happy path or something breaks. Sometimes it can be a long while before that happens, sometimes it happens right away.

Re: SQLite is all you need for durable workflows

#187

Post by petcat »

nemothekid wrote: post #117
petcat wrote: post #84

Earlier quoted context omitted.

Woof. That sounds very complicated. If you need that kind of write concurrency, use an unlogged table in postgres [0]. Then you don't have to invent a whole sharded thing yourself. [0] https://www.postgresql.org/docs/current/sql-createtable.html...

There are so many unfortunate footguns with unlogged tables, that I'd argue that the goroutine route is preferable.

What are the "footguns" with unlogged tables in Postgres?

Re: SQLite is all you need for durable workflows

#188

Post by kellogah »

therealdrag0 wrote: post #161
MattJ100 wrote: post #143

Earlier quoted context omitted.

You can do big data in SQLite. Concurrent writes, sure, I'd recommend something else. If you think the majority of systems require massively concurrent writes, I think you need to look a bit harder. SQLite is, after all, the most widely deployed database system, ever.

It’s widely deployed as a local DB for local apps like phones, desktops, and web browsers. But it’s not the most used for distributed, concurrent web apps, which db servers were designed for. Maybe people are talking past each other, but that’s the debate I see.

Not sure why there’s a debate at all. The discussion is on using SQLite instead of jq and markdown files. People got lost on a tangent! :)

Re: SQLite is all you need for durable workflows

#189

Post by kellogah »

abtinf wrote: post #58
levkk wrote: post #22

I don't understand this obsession with SQLite for real, production apps. SQLite is an embedded database, completely unsuitable for managing concurrency. This is what database _servers_ are for, e.g., Postgres, MySQL, etc. Their entire job is to allow you to modify data from multiple processes, on different machines, at the same time. This is a foundational principle of computer science. It seems to me that the "SQLit…

There are many cases where SQLite + concurrent front end (like a go net/http server) can handle all the load that a service might ever conceivably have to handle, especially if allowed to scale up hardware over time. You can trivially scale up SQLite to, what, hundreds of thousands of tps? The only thing you really give up is HA/failover and DR. But there are solutions to deal with those. And single-server systems ar…

I was thinking of using SQLite on top of k3s/Longhorn to replicate it. Anyone do something similar? Folks mention light steam and aws but Jeff Bezos’s biceps are too much for me to handle.

Re: SQLite is all you need for durable workflows

#190

Post by gnabgib »

kellogah wrote: post #188
therealdrag0 wrote: post #161

Earlier quoted context omitted.

It’s widely deployed as a local DB for local apps like phones, desktops, and web browsers. But it’s not the most used for distributed, concurrent web apps, which db servers were designed for. Maybe people are talking past each other, but that’s the debate I see.

Not sure why there’s a debate at all. The discussion is on using SQLite instead of jq and markdown files. People got lost on a tangent! :)

No it's not.. the context of other threads on this post (which mention jq) do not apply here. How poorly coded are you?
Post reply on HN