Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite is all you need for durable workflows

obeli.sk

Post reply on HN
Print view View original

351–360 of 413 posts

Re: SQLite is all you need for durable workflows

#351

Post by pluralmonad »

refulgentis wrote: post #179
wat10000 wrote: post #154

Earlier quoted context omitted.

It's very likely that you have multiple SQLite databases in your pocket right now. It's one of the most widely deployed pieces of software on the planet. If your conclusion is that it's guaranteed to cause more problems than other solutions, then that's on you.

Correct! I'm not "worried" about it, I've been putting SQLites in your and my pocket for the last 17 years. I don't want to be glib and leave it there, even though I'm slightly annoyed you missed several sigils in my post that I was well past that. The point is, for the not in your pocket case, for the not a singular document store case, I'm curious what the use case is.

I use it to keep infra spend low for some systems I built/maintain for a handful of volunteer orgs. These systems have multiple users, dozens to a couple hundred. I just serialize writes in app code. Backup the db files to blob storage every so often and don't think about it much more.

Re: SQLite is all you need for durable workflows

#352

Post by UltraSane »

levkk wrote: post #22

I don't understand this obsession with SQLite for real, production apps. SQLite is an embedded database, completely unsuitable for managing concurrency. This is what database _servers_ are for, e.g., Postgres, MySQL, etc. Their entire job is to allow you to modify data from multiple processes, on different machines, at the same time. This is a foundational principle of computer science. It seems to me that the "SQLit…

SQLite also gets really slow at around 50 million rows.

Re: SQLite is all you need for durable workflows

#353

Post by pseudalopex »

formerly_proven wrote: post #293
pseudalopex wrote: post #203

Earlier quoted context omitted.

The price of compatibility could be a pragma.

It literally is? Changing the defaults shown in the PRAGMAs above would break backwards compatibility. SQLite is strictly semantically versioned and does not break backwards compatibility. https://sqlite.org/versionnumbers.html

Their complaint was SQLite's type system was poor. You said this was the price of compatibility. And the documentation of your recommended pragmas said nothing of types. They seemed unrelated helpful advice seemingly.

New types would break forward compatibility in SQLite's terms. 3.7.0 added WAL mode was their example of a forward compatibility break.[1] 3.y.0 could add better type system mode.

[1] https://sqlite.org/formatchng.html

Re: SQLite is all you need for durable workflows

#354

Post by nemothekid »

petcat wrote: post #187
nemothekid wrote: post #117

Earlier quoted context omitted.

There are so many unfortunate footguns with unlogged tables, that I'd argue that the goroutine route is preferable.

What are the "footguns" with unlogged tables in Postgres?

1. If postgres shutsdowns uncleanly, your entire table is truncated; you lose everything.

2. You should check if your backup method backs up unlogged tables. For example, RDS Snapshots on AWS do not backup unlogged tables.

These 2 are a double whammy where if you aren't aware of these tradeoffs you can find that a bad restart has deleted all your data, plus your unlogged tables were never backed up.

Re: SQLite is all you need for durable workflows

#355

Post by tomasol »

oulipo2 wrote: post #350
sgloutnikov wrote: post #6

It's close enough that DBOS does support SQLite. [0] The default for prototyping is SQLite, but sure you can run it in production if you wanted. Obligatory list of workflow engines and libraries because it's such a common need that a lot have rolled their own. [1] [0] https://docs.dbos.dev/python/tutorials/database-connection [1] https://github.com/meirwah/awesome-workflow-engines

What would be the main differences between DBOS and Obelisk?

Hi, I wrote a comparison blog post between Obelisk and the Java version of DBOS a couple of months ago: https://obeli.sk/blog/comparing-dbos-part-1/

Re: SQLite is all you need for durable workflows

#356

Post by tomasol »

oulipo2 wrote: post #348

Obelisk also supports Postgres. When you're using it with Postgres, what are the differences with DBOS? Are there any that would be significant (I'm already using Postgres, so I don't really need a sqlite-based durable workflow engine, so looking to know how to choose between DBOS and Obelisk)

Check out https://obeli.sk/blog/comparing-dbos-part-1/

Re: SQLite is all you need for durable workflows

#359

Post by inglor »

UnfitFootprint wrote: post #316
inglor wrote: post #266

Earlier quoted context omitted.

In the last two years, we built (with a team of 15, now 100) a billion dollar business on top of Temporal that performs business critical applications for fortune 500 companies. We couldn't be happier with temporal. Determinism sucks, you do have to work hard and make everything idempotent in activities like we would for durable software anyway. The language we used was incorrect (Go) and has a lot of boilerplate com…

Could you share a bit more about your learnings on go + temporal? That combo was next in line for us to migrate _to_

Sure, basically:

- Temporal itself is written in Go and we use Go for our backend so we expected this to be a natural fit. - Temporal makes writing activities in Go very explicit and boilerplatey - This in turn makes testing more difficult than it needs to be often - Temporal doesn't play well with Go's concurrency model at all (all stuff like goroutines needs to go through its special workflows.Go) a lot more often you have to write stuff that "appeases" temporal. - The whole workflows.ExecuteActivity(...).Get(...) is weird, having futures in a language explcitly designed to avoid that is weird. - All our compute isn't done on temporal workers anyway, its done (in another AWS account, owned by the customer) in batch compute (aws batch, lambda, ec2, whatever) so our temporal code isn't CPU heavy but is highly concurrent and needs a very high reliability guarantee. - Compare that to temporal with TypeScript, where it's simple and easy to use the same code inside or outside of temporal. Testing is trivial and the code looks like "regular code".

Re: SQLite is all you need for durable workflows

#360

Post by inglor »

taberiand wrote: post #272
fcarraldo wrote: post #207

Earlier quoted context omitted.

It sounds like you’re running this mostly on a single machine? Temporal gets much more complex with scale. Cassandra isn’t fun to manage. Ringpop and TChannel are hard to debug when things go wrong. The SQL backend support doesn’t support horizontally scaled replicas (just single instance) due to consistency requirements. Depending on how your code is written, modifying code baked into workflows becomes complex, as a…

> Depending on how your code is written, modifying code baked into workflows becomes complex, as anything that modifies the history event ordering breaks determinism in already-deployed workers. I see this as Temporal surfacing inherent complexity of the domain in a way that forces the developer to consider it, rather than introducing extra complexity. If it didn't make workflow determinism a strict requirement, the…

Actually Temporal does have a way to avoid determinism called rainbow deployments.

If you're fine with deploying several versions of workers (and are on a reasonably new version) you can just avoid the determinism issue altogether with their k8s controller.

If you do need to have some long workflows, there is an explicit hook for "what happens to existing workflows on version upgrade".

But to be fair - none of the other orchastrators I used (like AirFlow) made me write workflows.IsNewCode/IsOldCode like temporal does. On the other hand AirFlow doesn't even have the capability to do that in the first place (or at least it didn't last I used it).

Post reply on HN