Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Durable execution should be lightweight

dbos.dev

Post reply on HN
Print view View original

41–45 of 45 posts

Re: Durable execution should be lightweight

#41

Post by com2kid »

peterkelly wrote: post #8

Durable execution is best done at the level of a language implementation, not as a library. A workflow engine I recently built provided an interpreter for a Scheme-based language that, for each blocking operation, took a snapshot of the interpreter state (heap + stack) and persisted that to a database. Each time an operation completes (which could be after hours/days/weeks), the interpreter state is restored from the…

This comment reminds me that despite having built lots of cool stuff in my career, that in reality I've just barely done anything that can be called "software engineering".

Re: Durable execution should be lightweight

#42

Post by ethbr1 »

KraftyOne wrote: post #31
ethbr1 wrote: post #21

Earlier quoted context omitted.

You're forcing adopters to divide any state-impactful activity into its own function (because only functions can be decorated with step, no?). That's seriously inelegant when scaled to larger codebases. Regional tagging (e.g. safe/unsafe) would be a better approach, as it would allow developers to more naturally protect code, without redefining its structure to suit your library. You start to grok the problem here, b…

Yeah, this definitely requires splitting state-impactful activity into its own function. That's good practice anyways, though I understand it might be a pain in large codebases. Regional tags are definitely an interesting alternative! For the UI example, I don't think you'd use durable execution for most of the UI--it's just not needed. But maybe there's one button that launches a complex asynchronous background task…

> splitting state-impactful activity into its own function.

Each separate state-impactful activity into its own function, no?

> For the UI example...

I wasn't talking about building a GUI in a python program: I was talking about using python to automate an external GUI app.

Because that toy example (which also happens to be done in the real world!) encapsulates a lot of the state issues that don't seem well-handled by the current design.

Namely, that external interactions are a mix between stateless and stateful operations.

Re: Durable execution should be lightweight

#43

Post by octonaut »

KraftyOne wrote: post #19
shipp02 wrote: post #4

This seems like temporal only without as much server and complexity. Maybe they ignore it or it really is that simple. Overall really cool! There are some scalability concerns that are brought that I think are valid but maybe you have a Postgres server backing up every few servers that need this kind of execution. Also, every function shouldn't be its own step but needs to be divided into larger chunks where every re…

Thanks! DBOS is simpler not because it ignores complexity, but because it uses Postgres to deal with complexity. And Postgres is a very powerful tool for building reliable systems!

Temporal has the option of using postgres as the persistence backend. Presumably, the simplicity of DBOS comes from not having to spin up a webserver and workflow engine to orchestrate the functions?

Re: Durable execution should be lightweight

#44

Post by haimzlato »

peterkelly wrote: post #8

Durable execution is best done at the level of a language implementation, not as a library. A workflow engine I recently built provided an interpreter for a Scheme-based language that, for each blocking operation, took a snapshot of the interpreter state (heap + stack) and persisted that to a database. Each time an operation completes (which could be after hours/days/weeks), the interpreter state is restored from the…

We (AutoKitteh) took a different approach to provide durable execution at the language level: We are using Temporal as a durable execution engine, but we built a layer on top of it that provides a pure Python experience (JS coming soon). This is done by hooking into the AST and converting functions that might have side effects to Temporal activities. Of course it's deeper than that.

For the user, it's almost transparent. Regular Python code (no decorators) that you can run as Duralbe workflow in our system or as regular Python.

You can look at the open-source, and we have a blog explaining how it's implemented.

Re: Durable execution should be lightweight

#45

Post by shayansm1 »

> In some sense, external orchestration turns individual applications into distributed microservices, with all the complexity that implies.

While I'm not entirely convinced by the notion that distributed microservices inherently increase complexity, I do see significant benefits in how they empower workflows that span multiple projects and teams. For instance, in Temporal, different workers can operate in various programming languages, each managing its own specific set of activities within a single workflow. This approach enhances communication and collaboration between diverse projects, allowing them to leverage their unique tech stacks while still working together seamlessly.

Post reply on HN