Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Durable execution should be lightweight

dbos.dev

Post reply on HN
Print view View original

21–30 of 45 posts

Re: Durable execution should be lightweight

#21

Post by ethbr1 »

KraftyOne wrote: post #17
ethbr1 wrote: post #15

Hot take: this is bad architecture. The solution seems to be solving for the simplest use case (internal stateless functions) rather than the most complex use case (external state-impactful functions). Furthermore, the words used aren't really what they should be talking about. >> Because workflows are just Python functions, the thread can restart a workflow by simply calling the workflow function with its original i…

That's exactly what this model is! The @Step decorator is for external state modifications. Then @Workflows orchestrate steps. The example shows the simplest possible external state modification--a print to the terminal. Steps can be tried multiple times (if a failure happens mid-step) but never re-execute once complete. Since idempotency can't be added externally, that's the strongest possible guarantee any orchestr…

You're forcing adopters to divide any state-impactful activity into its own function (because only functions can be decorated with step, no?). That's seriously inelegant when scaled to larger codebases.

Regional tagging (e.g. safe/unsafe) would be a better approach, as it would allow developers to more naturally protect code, without redefining its structure to suit your library.

You start to grok the problem here, but primarily think about it in terms of databases, which are just one (admittedly common) type of external state:

>> If you need to perform a non-deterministic operation like accessing the database, calling a third-party API, generating a random number, or getting the local time, you shouldn't do it directly in a workflow function. Instead, you should do all database operations in transactions and all other non-deterministic operations in steps.

Note: Think you should really change "all" into "each in a separate transaction/step" there, to communicate what you're recommending?

As a thought exercise: imagine a Python program that automates a third party application via the GUI. Some UI actions cannot be undone (e.g. submit). Some are repeatable without consequence (e.g. navigating between screens).

How would your framework support that?

Because if you can efficiently support the pathological leaky-state case, you can trivially support all simpler cases.

Re: Durable execution should be lightweight

#22

Post by CGamesPlay »

Love it! I built a toy library that looked very similar to this one a few months ago. How does this handle changing the workflow code? I quite like how Temporal handles it, where you use an "if has(my_feature)" to allow for in-progress workflows to be live-updated, even in the middle of loops. I also introduced an idea of "object handles", something like a file descriptor, which is an opaque handle to the workflow function but which can be given to a step function to be unwrapped, and it can can be persisted and restored via a consistent ID.

Re: Durable execution should be lightweight

#23

Post by sandruso »

peterkelly wrote: post #8

Durable execution is best done at the level of a language implementation, not as a library. A workflow engine I recently built provided an interpreter for a Scheme-based language that, for each blocking operation, took a snapshot of the interpreter state (heap + stack) and persisted that to a database. Each time an operation completes (which could be after hours/days/weeks), the interpreter state is restored from the…

I was playing with resumable execution and I agree that language-level support would be immense improvement. That said, I was able create library that would take javascript code and execute it line-by-line where after each line the state is stored but user needs to use state object to store all intermediate results which are persisted. But there is still problems if the computation fails during on some line. We can retry it but if it broke the entire execution state then we have a problem.

I wonder how to achieve the transaction-like (commit/rollback) behavior that will works across boundaries. Doing it on language-level is the way. Compiler/interpreter can handle all the state serializations.

Re: Durable execution should be lightweight

#24

Post by necovek »

peterkelly wrote: post #8

Durable execution is best done at the level of a language implementation, not as a library. A workflow engine I recently built provided an interpreter for a Scheme-based language that, for each blocking operation, took a snapshot of the interpreter state (heap + stack) and persisted that to a database. Each time an operation completes (which could be after hours/days/weeks), the interpreter state is restored from the…

As soon as you do that, you tie in your state-preserving storage (a database?) with your language as your "programming environment", and it becomes harder to decouple them (or the design becomes overly complex with configurable implementations of a state-database interface).

So, I don't think this should be at the language level. Potentially at a programming environment level, which includes configuration for such environment, but separation of concerns is screaming loudly in my head :)

Re: Durable execution should be lightweight

#25

Post by movedx01 »

jkonowitch wrote: post #10
zokier wrote: post #6

My immediate reaction is hell no. > In some sense, external orchestration turns individual applications into distributed microservices, with all the complexity that implies. I'd argue that durable execution intrisically is complex and external orchestrators give you tools to manage that complexity, whereas this attempts to brush the complexity under the rug in a way that does not inspire confidence.

What value does an external orchestrator add for managing complexity that an “embedded” solution could not?

From my experience, the embedded solution becomes a distributed orchestrator and, thus, is no longer simpler.

Re: Durable execution should be lightweight

#26

Post by from-nibly »

jkonowitch wrote: post #10
zokier wrote: post #6

My immediate reaction is hell no. > In some sense, external orchestration turns individual applications into distributed microservices, with all the complexity that implies. I'd argue that durable execution intrisically is complex and external orchestrators give you tools to manage that complexity, whereas this attempts to brush the complexity under the rug in a way that does not inspire confidence.

What value does an external orchestrator add for managing complexity that an “embedded” solution could not?

Where is the state stored? What if I have 5 copies of my service now, and 3 later. What happens to the "embedded state"? Was it on the container? Did it just evaporate? Was it on a volume? Is it stuck in limbo? Even if it was put into s3 it would be stuck in limbo.

Re: Durable execution should be lightweight

#27

Post by from-nibly »

This is the can of worms introduced by doing event processing. As soon as you break the ties between request and response a billion questions come up. Asking what happens to the reservation in a request response scenario is just chuck an error at the user and ask them to try again.

As soon as you accept the user input and tell the user all is well before you have processed it you enter into these kinds of problems.

I know not every single thing we do can be done without this kind of async processing, but we should treat these scenarios more seriously.

It's not that it can't ever be good, it's that it will always be complicated.

Re: Durable execution should be lightweight

#28

Post by magicalhippo »

peterkelly wrote: post #8

Durable execution is best done at the level of a language implementation, not as a library. A workflow engine I recently built provided an interpreter for a Scheme-based language that, for each blocking operation, took a snapshot of the interpreter state (heap + stack) and persisted that to a database. Each time an operation completes (which could be after hours/days/weeks), the interpreter state is restored from the…

Reminded me of the Restate idea[1], discussed here[2] recently, except they do it as a library. An excerpt:

To persist intermediate steps (line 8), handlers use the SDK (ctx.run), which sends the event to the log and awaits the ack of the conditional append to the event’s execution journal. On retries, the SDK checks the journal whether the step’s event already exists and restores the result from there directly.

Though I think I agree with your point that it would be better to have this even more integrated than "just" a library. Perhaps something like how you can override the global memory allocator in C and similar languages, to avoid a tight coupling to the persistence layer.

[1]: https://restate.dev/blog/every-system-is-a-log-avoiding-coor...

[2]: https://news.ycombinator.com/item?id=42813049

Re: Durable execution should be lightweight

#29

Post by KraftyOne »

from-nibly wrote: post #26
jkonowitch wrote: post #10

Earlier quoted context omitted.

What value does an external orchestrator add for managing complexity that an “embedded” solution could not?

Where is the state stored? What if I have 5 copies of my service now, and 3 later. What happens to the "embedded state"? Was it on the container? Did it just evaporate? Was it on a volume? Is it stuck in limbo? Even if it was put into s3 it would be stuck in limbo.

DBOS solves this problem by storing state in Postgres, which is really good at coordinating multiple copies of the same service. Essentially, Postgres does the hard parts of external orchestration, letting you work with a simple library abstraction.

Re: Durable execution should be lightweight

#30

Post by KraftyOne »

from-nibly wrote: post #27

This is the can of worms introduced by doing event processing. As soon as you break the ties between request and response a billion questions come up. Asking what happens to the reservation in a request response scenario is just chuck an error at the user and ask them to try again. As soon as you accept the user input and tell the user all is well before you have processed it you enter into these kinds of problems. I…

Agreed, if you can do something completely synchronously while responding to an HTTP request, you should.

But often you can't! Then, durable execution helps you manage the complexity of async processing.

Post reply on HN