Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Restate – Low-latency durable workflows for JavaScript/Java, in Rust

restate.dev

Post reply on HN
Print view View original

41–50 of 112 posts

Re: Show HN: Restate – Low-latency durable workflows for JavaScript/Java, in Rust

#41

Post by rubyfan »

p10jkle wrote: post #34
rubyfan wrote: post #29

There’s a lot of jargon in this, is there a lay person explanation of what problem this solves?

Our goal is to make it easier to write code that handles failures - failed outbound api calls, infrastructure issues like a host dying, problems talking between services. The primitive we offer is that we guarantee that your handlers always run to completion (whether to a result or a terminal error) The way we do that is by writing down what your code is doing, while its doing it, to a store. Then, on any failure, we…

So non-response time bound workloads that need to reliably dispatch other processes to completion?

Would a good example be something like, automated highway toll collecting? i.e. I drive past a scanner on the highway, my license plate is scanned and several state bound collection events need to be triggered until the toll is ultimately collected?

Re: Show HN: Restate – Low-latency durable workflows for JavaScript/Java, in Rust

#42

Post by delusional »

p10jkle wrote: post #34
rubyfan wrote: post #29

There’s a lot of jargon in this, is there a lay person explanation of what problem this solves?

Our goal is to make it easier to write code that handles failures - failed outbound api calls, infrastructure issues like a host dying, problems talking between services. The primitive we offer is that we guarantee that your handlers always run to completion (whether to a result or a terminal error) The way we do that is by writing down what your code is doing, while its doing it, to a store. Then, on any failure, we…

> where the code doesn't have to be idempotent

Is that true? I don't think that makes any theoretical sense, since I'm pretty sure the whole thing relies on transparent retries for external calls.

If I complete some action that can't be retried and then die before writing it to the log (completing an action unatomically) there would seem to be no way for this to recover without idempotency.

Re: Show HN: Restate – Low-latency durable workflows for JavaScript/Java, in Rust

#43

Post by p10jkle »

delusional wrote: post #35
p10jkle wrote: post #31

Earlier quoted context omitted.

not necessarily - we store the intermediary states of your handler, so it can be replayed on infrastructure failures. if the handler changes in what it does, those intermediary states (the 'journal') might no longer match this. the best solution is to route replayed requests to the version of the code that originally executed the request, but: 1. many infra platforms dont allow you to execute previous versions 2. aft…

I was of course just thinking about the "front" of the execution, when you're sleeping for 2 days and you want to switch out a future step. Switching out logic that has already been committed is a harder problem. That's a goo point. > after some duration (maybe just minutes), executing old code is dangerous, eg because of insecure dependencies. Could you elaborate on that? My understanding is that all of this tech bu…

> I was of course just thinking about the "front" of the execution, when you're sleeping for 2 days and you want to switch out a future step. Switching out logic that has already been committed is a harder problem. That's a goo point.

You make a good point - this is the idea behind 'delayed calls' which are really one of my favourite things about Restate. Don't save all the intermediary state - just serialise the service name, the handler name, and the arguments, and store that for a month or whatever. That is a very tractable problem - ie just request object versioning

Re: Show HN: Restate – Low-latency durable workflows for JavaScript/Java, in Rust

#44

Post by p10jkle »

corytheboyd wrote: post #39
p10jkle wrote: post #34

Earlier quoted context omitted.

Our goal is to make it easier to write code that handles failures - failed outbound api calls, infrastructure issues like a host dying, problems talking between services. The primitive we offer is that we guarantee that your handlers always run to completion (whether to a result or a terminal error) The way we do that is by writing down what your code is doing, while its doing it, to a store. Then, on any failure, we…

What if the code was changed by the time it is retried? I imagine it would have to throw away its memorized instructions, and because the code isn’t idempotent…

Great question! https://news.ycombinator.com/item?id=40659687

Re: Show HN: Restate – Low-latency durable workflows for JavaScript/Java, in Rust

#45

Post by aleksiy123 »

Looks really awesome. Always been looking for some easy to use async workflows + cronjobs service to use with serverless like Vercel.

Also something about this area always makes me excited. I guess it must be the thought of having all these tasks just working in the background without having to explicitly manage them.

One question I have is does anyone have experience for building data pipelines in this type of architecture?

Does it make sense to fan out on lots of small tasks? Or is it better to batch things into bigger tasks to reduce the overhead.

Re: Show HN: Restate – Low-latency durable workflows for JavaScript/Java, in Rust

#46

Post by johtso »

p10jkle wrote: post #34
rubyfan wrote: post #29

There’s a lot of jargon in this, is there a lay person explanation of what problem this solves?

Our goal is to make it easier to write code that handles failures - failed outbound api calls, infrastructure issues like a host dying, problems talking between services. The primitive we offer is that we guarantee that your handlers always run to completion (whether to a result or a terminal error) The way we do that is by writing down what your code is doing, while its doing it, to a store. Then, on any failure, we…

Doesn't anything involving requests to other services inherently have to be idempotent because there's still a chance of a communication error resulting in an unknown outcome of the action? You don't know if the "widget order" was successfully placed or not, and therefore there's no way to know if that action can safely be tried again.

Re: Show HN: Restate – Low-latency durable workflows for JavaScript/Java, in Rust

#48

Post by p10jkle »

delusional wrote: post #42
p10jkle wrote: post #34

Earlier quoted context omitted.

Our goal is to make it easier to write code that handles failures - failed outbound api calls, infrastructure issues like a host dying, problems talking between services. The primitive we offer is that we guarantee that your handlers always run to completion (whether to a result or a terminal error) The way we do that is by writing down what your code is doing, while its doing it, to a store. Then, on any failure, we…

> where the code doesn't have to be idempotent Is that true? I don't think that makes any theoretical sense, since I'm pretty sure the whole thing relies on transparent retries for external calls. If I complete some action that can't be retried and then die before writing it to the log (completing an action unatomically) there would seem to be no way for this to recover without idempotency.

Absolutely, individual atomic side effects need to be idempotent. We can't solve the fundamental distributed system problem there (eg an HTTP 500 - did it actually get executed) However, the string of operations doesn't need to be idempotent - lets say your handler does 3 tasks A B C, and the machine dies at C. Only C will be re-executed. A and B need to be atomically idempotent, but once we move on, we don't start again

Critical point - its much easier to think about and test for the re-execution of C in a vacuum, than to test for A B C all re-executing in sequence, with a variable number of those having already executed before

Re: Show HN: Restate – Low-latency durable workflows for JavaScript/Java, in Rust

#49

Post by p10jkle »

johtso wrote: post #46
p10jkle wrote: post #34

Earlier quoted context omitted.

Our goal is to make it easier to write code that handles failures - failed outbound api calls, infrastructure issues like a host dying, problems talking between services. The primitive we offer is that we guarantee that your handlers always run to completion (whether to a result or a terminal error) The way we do that is by writing down what your code is doing, while its doing it, to a store. Then, on any failure, we…

Doesn't anything involving requests to other services inherently have to be idempotent because there's still a chance of a communication error resulting in an unknown outcome of the action? You don't know if the "widget order" was successfully placed or not, and therefore there's no way to know if that action can safely be tried again.

https://news.ycombinator.com/item?id=40659968 Absolutely, sorry if im not tight enough with my language. Maybe should be described as 'operation idempotency' vs 'handler idempotency'. IMO, an entire handler re-executing is much harder to reason about and test for than a particular operation re-executing individually, with nothing else changing between executions

Re: Show HN: Restate – Low-latency durable workflows for JavaScript/Java, in Rust

#50

Post by stsffap »

jamifsud wrote: post #47

Any plans for a Python SDK? We’re actively looking for a platform like this but our stack is TS / Python!

We are actively looking for feedback on what SDK to develop next. Quite a few people have voiced interest in Python so far. This will make it more likely that we might tackle this soonish. We'll keep you posted.
Post reply on HN