Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Make any TypeScript function durable

useworkflow.dev

Post reply on HN
Print view View original

81–89 of 89 posts

Re: Make any TypeScript function durable

#81

Post by yencabulator »

MrJohz wrote: post #69
halflife wrote: post #15

Earlier quoted context omitted.

You can use generator fns to achieve the exact same thing without magic strings or bundles. And the bonus is that you can debug it.

I don't believe you can. I believe what they're trying to do is rewrite the underlying function into separate functions that can be called in any order and combination. That's not possible with generators. With a generator, I can pause between steps, but I can't, say, retry the third step four times until it succeeds, suspend the entire process, and then jump straight to the fourth step. Or I can't run different step…

> With a generator, I can pause between steps, but I can't, say, retry the third step four times until it succeeds, suspend the entire process, and then jump straight to the fourth step.

If I recall correctly, other solutions in this space work by persisting & memoizing the results of the steps as they succeed, so the whole thing can be rerun and anything already completed uses the memoized result.

Re: Make any TypeScript function durable

#82

Post by nsonha »

MrJohz wrote: post #14
Etheryte wrote: post #2

Of all the syntax options they could've gone with, they settled on what I would say is arguably the worst. If you want a one-liner, decorators are widely used across different languages and Typescript supports them as well.

Decorators don't work for functions, unfortunately, so wouldn't work in this case. You'd need to add a bunch of class boilerplate to make it work. JavaScript didn't have a lot of great options for this kind of statically-declared metaprogramming, which is why the "use X"; syntax has become so popular in various spaces. It's definitely not my favourite approach, but I don't think there's any clear "best" solution here…

bunch of class "boilerplate" is 2 lines to wrapping the thing in a class, which is most likely the right thing to do anyway. You would want to group the durable functions and manage the dependencies in some way.

Re: Make any TypeScript function durable

#83

Post by Spivak »

Every durable function platform is so disappointing because they all try to work without solving the hard problem: serializing, snapshotting, and restoring a running program. They all have some API that pushes the work of state management and safepoints on to you the developer.

Once you have the primitive of real durable compute all the hard bits fall away. And it's not as if this is some fantasy, VM live migration is a real working example of it working. Then you just write your program in the grug way, use your language's built in retry tools and store state in normal variables because the entire thing is durable including memory, cpu state, gpu state, network state, open files, etc..

Re: Make any TypeScript function durable

#84

Post by dtornow »

halflife wrote: post #15
MrJohz wrote: post #14

Earlier quoted context omitted.

Decorators don't work for functions, unfortunately, so wouldn't work in this case. You'd need to add a bunch of class boilerplate to make it work. JavaScript didn't have a lot of great options for this kind of statically-declared metaprogramming, which is why the "use X"; syntax has become so popular in various spaces. It's definitely not my favourite approach, but I don't think there's any clear "best" solution here…

You can use generator fns to achieve the exact same thing without magic strings or bundles. And the bonus is that you can debug it.

At Resonate, we are using generators (both for the typescript sdk and the python sdk) to implement Distributed Async Await, Resonate's Durable Execution Framework. Because generators transfer control back to the caller, you can essentially build your own event loop and weave distributed coordination (RPCs and callbacks) and distributed recovery (restarts) right into the execution with fairly little lines of code.

Disclaimer: I'm the CEO of Resonate

Re: Make any TypeScript function durable

#85

Post by dtornow »

yencabulator wrote: post #81
MrJohz wrote: post #69

Earlier quoted context omitted.

I don't believe you can. I believe what they're trying to do is rewrite the underlying function into separate functions that can be called in any order and combination. That's not possible with generators. With a generator, I can pause between steps, but I can't, say, retry the third step four times until it succeeds, suspend the entire process, and then jump straight to the fourth step. Or I can't run different step…

> With a generator, I can pause between steps, but I can't, say, retry the third step four times until it succeeds, suspend the entire process, and then jump straight to the fourth step. If I recall correctly, other solutions in this space work by persisting & memoizing the results of the steps as they succeed, so the whole thing can be rerun and anything already completed uses the memoized result.

That's what we do at Resonate: We build Distributed Async Await, basically async await with durability provided by Durable Promises. A Durable Promises is the checkpoint (the memoization device). When the generator restarts, the generator skips what has already been done.

We don't have workflow and steps tho, like async await, just functions all the way down.

Disclaimer: I'm the CEO of resonate

Re: Make any TypeScript function durable

#86

Post by dtornow »

MrJohz wrote: post #76
halflife wrote: post #70

Earlier quoted context omitted.

You are partly correct, if each step is another generator function you can retry the steps until success or fail, and even create a small framework around it.

That still requires the process running the outer generator to stay active the entire time. If that fails, you can't retry from step three. You need some way of starting the outer function part way through. You can do that manually by defining a bunch of explicit steps, their outputs and inputs, and how to transform the data around. And you could use generators to create a little DSL around that. But I think the idea…

When we built Distributed Async Await, we went a step further: Every time the generator instance awaits, we "kill" the generator instance (you cannot really kill a generator, but you can just let it go out of scope) and create a new one when the awaited computation completes. So in essence, we built resume semantics on top of restart. We were inspired by the paper Crash Only Software https://dslab.epfl.ch/pubs/crashonly.pdf

Re: Make any TypeScript function durable

#87

Post by halflife »

dtornow wrote: post #84
halflife wrote: post #15

Earlier quoted context omitted.

You can use generator fns to achieve the exact same thing without magic strings or bundles. And the bonus is that you can debug it.

At Resonate, we are using generators (both for the typescript sdk and the python sdk) to implement Distributed Async Await, Resonate's Durable Execution Framework. Because generators transfer control back to the caller, you can essentially build your own event loop and weave distributed coordination (RPCs and callbacks) and distributed recovery (restarts) right into the execution with fairly little lines of code. Dis…

This is pretty funny, I’m the architect at Rezonate

Re: Make any TypeScript function durable

#88

Post by tomp1998 »

trevor-e wrote: post #39

So this seems similar to https://temporal.io/ , am I reading this right? I used that briefly a few years ago and it was pretty nice at the time. It did make some features much easier to model like their welcome email example. Would love to hear from someone with extensive temporal experience, iirc the only drawback was on the infra side of things.

[dead]

Re: Make any TypeScript function durable

#89

Post by slinkydeveloper »

dangoodmanUT wrote: post #53
rubenfiszel wrote: post #41

Earlier quoted context omitted.

workflows is just short for state machine DSL

but with history length and replay determinism landmines

Not sure why these not issues with state machines...

With RSM you often need to keep the history as well, just in a different form, that is the message/event log that your state machine processed.

With RSM you need your code to be deterministic to rebuild the materialized state during failover, unless you snapshot and replicate the state on each processed event/message.

Post reply on HN