Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite is all you need for durable workflows

obeli.sk

Post reply on HN
Print view View original

261–270 of 413 posts

Re: SQLite is all you need for durable workflows

#261

Post by stingraycharles »

franga2000 wrote: post #239
levkk wrote: post #22

I don't understand this obsession with SQLite for real, production apps. SQLite is an embedded database, completely unsuitable for managing concurrency. This is what database _servers_ are for, e.g., Postgres, MySQL, etc. Their entire job is to allow you to modify data from multiple processes, on different machines, at the same time. This is a foundational principle of computer science. It seems to me that the "SQLit…

And I don't understand the obsession with server-based databases for single apps. Especially in containerised setups, every "app" gets its own database anyways, and if the app is further broken down into services, they usually communicate between each other and not with a shared database. So in those cases, what do you gain by pulling the database out of the "process" and onto the other end of a socket? In most cases…

“Especially in containerised setups, every "app" gets its own database anyways, and if the app is further broken down into services, they usually communicate between each other and not with a shared database. “

You seem to be talking about a vastly different use case.

Containerized apps having their own database? What? Aren’t these types of containers stateless? I always very much try to keep state out of app containers.

What kind of data storage are we talking about?

Re: SQLite is all you need for durable workflows

#264

Post by Gud »

sharts wrote: post #251
MattJ100 wrote: post #143

Earlier quoted context omitted.

You can do big data in SQLite. Concurrent writes, sure, I'd recommend something else. If you think the majority of systems require massively concurrent writes, I think you need to look a bit harder. SQLite is, after all, the most widely deployed database system, ever.

Internet Explorer 6 was the most widely deployed awesome piece of software. Those that hated it need to look a bit harder.

It was not really “deployed” by a lot of people in the same sense.

It was forced upon most of us(not me, I used BeOS then Debian then FreeBSD).

I deployed phoenix.

Re: SQLite is all you need for durable workflows

#265

Post by utopiah »

The cycle of expertise :

- what is X, I just do Y

- wow I can see so many limits of Y, now I do X

- I use X for literally everything

- now that I properly understand the limits of Y but also the heavy constraints of X ... maybe Y is enough

- I use Y for literally everything

rinse & repeat. The thing is with actual usage and actual context one does learn and thus can get away with a lot more "basic" solution but it does require genuinely understanding the limits.

Re: SQLite is all you need for durable workflows

#266

Post by inglor »

fcarraldo wrote: post #207
bitexploder wrote: post #7

I started setting up my workflows using Temporal. It deploys as relatively light weight local app. For an isolated local installation it uses SQLite. It makes the process of dealing with API retries and organizing workflows and tasks really simple. I recommend giving it a try. It is, philosophically, exactly what this article is suggesting, but it adds an incredibly rich and flexible interface for agents to work with…

It sounds like you’re running this mostly on a single machine? Temporal gets much more complex with scale. Cassandra isn’t fun to manage. Ringpop and TChannel are hard to debug when things go wrong. The SQL backend support doesn’t support horizontally scaled replicas (just single instance) due to consistency requirements. Depending on how your code is written, modifying code baked into workflows becomes complex, as a…

In the last two years, we built (with a team of 15, now 100) a billion dollar business on top of Temporal that performs business critical applications for fortune 500 companies. We couldn't be happier with temporal.

Determinism sucks, you do have to work hard and make everything idempotent in activities like we would for durable software anyway. The language we used was incorrect (Go) and has a lot of boilerplate compared to alternatives we later investigated (Python and TypeScript). Visibility can be slow and misses information. We needed to write our own APIs to work effectively with Agents for root-cause analysis of failures.

With all the caveats - Temporal is amazing, it feels much better than previous orchestrators I used like Prefect or Airflow. 100% would adopt again.

Re: SQLite is all you need for durable workflows

#267

Post by dmos62 »

utopiah wrote: post #265

The cycle of expertise : - what is X, I just do Y - wow I can see so many limits of Y, now I do X - I use X for literally everything - now that I properly understand the limits of Y but also the heavy constraints of X ... maybe Y is enough - I use Y for literally everything rinse & repeat. The thing is with actual usage and actual context one does learn and thus can get away with a lot more "basic" solution but it do…

Persistence in folly leads a fool to enlightenment.

Re: SQLite is all you need for durable workflows

#268

Post by HDThoreaun »

franga2000 wrote: post #239
levkk wrote: post #22

I don't understand this obsession with SQLite for real, production apps. SQLite is an embedded database, completely unsuitable for managing concurrency. This is what database _servers_ are for, e.g., Postgres, MySQL, etc. Their entire job is to allow you to modify data from multiple processes, on different machines, at the same time. This is a foundational principle of computer science. It seems to me that the "SQLit…

And I don't understand the obsession with server-based databases for single apps. Especially in containerised setups, every "app" gets its own database anyways, and if the app is further broken down into services, they usually communicate between each other and not with a shared database. So in those cases, what do you gain by pulling the database out of the "process" and onto the other end of a socket? In most cases…

Every container gets its own database?

Re: SQLite is all you need for durable workflows

#269

Post by halamadrid »

Operators of Unmeshed here, which is basically a rewrite of Netflix Conductor. In this orchestrator we heavily use a uniquely scaled version of SQLite and also offers “managed” SQLite instances for managing user data. Combining the durable executions of Unmeshed and along with workflow primitives like sleep, workers, etc you can actually build complex systems with a lot less code than ever.

Check it out here: https://unmeshed.io

Re: SQLite is all you need for durable workflows

#270

Post by imhoguy »

faangguyindia wrote: post #214

I've replaced all of these with Go + SQLite: 1. Intercom 2. Zendesk 3. Email marketing 4. Kanban 5. Todo 6. Our billing stack 7. Our issue tracker 8. Our forum 9. Uptime monitor 10. PagerDuty (clone) I have dozens of products I sell, so I thought: why not build everything ourselves? All of these run on the same server and use very little memory. I replaced all the SaaS tools we used with these. I also moved to dedica…

Thanks for your words of wisdom. I share same sentiment of simple apps. I self-learned coding thru 1990s and I am sick of herding distributed overgrown software at work.

Quick question: how do you inspect database contents of your deployed (micro)apps? I used SQLite in one production app and what I didn't like was to use terminal sqlite client on the server or having to copy DB to my laptop to query it with sqlitebrowser. With my Postgres server it is much simpler to just query by rich GUI client with SSH tunnel.

Post reply on HN