Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Durable Coroutines for Go

github.com

Post reply on HN
Print view View original

11–20 of 37 posts

Re: Durable Coroutines for Go

#11

Post by achille-roussel »

skybrian wrote: post #9

The saved coroutine state is an implicitly-defined data structure and they don’t support any kind of migration when the code changes, so the durability will be quite limited.

You’re correct that code migration is still a to do, we’ll be exploring how to do that in the future.

As you can imagine there are a lot of challenges with it, but ideas are welcome!

Re: Durable Coroutines for Go

#12

Post by achille-roussel »

born-jre wrote: post #10

nice, but i had a separate idea. what if u build a wasm runtime that can save and restore memory with and execution states, sounds much more full proof. or i might have misunderstood this idea :D.

We tried that actually, and it can work well but you make a different set of trade offs. For example, you can’t get the granularity that durable coroutines give you, you’re bound to snapshot and restore the entire state of the application.

WASM is also not as mature of a tech for server side systems, a lot is left to figure out so the type of applications that you can build with it remains limited.

I’d be excited to see support for something like this get built tho!

Re: Durable Coroutines for Go

#13

Post by skybrian »

achille-roussel wrote: post #11
skybrian wrote: post #9

The saved coroutine state is an implicitly-defined data structure and they don’t support any kind of migration when the code changes, so the durability will be quite limited.

You’re correct that code migration is still a to do, we’ll be exploring how to do that in the future. As you can imagine there are a lot of challenges with it, but ideas are welcome!

It seems like generating some data structure definitions (similar to a protobuf) and checking them in along with the code would be useful? Then the compiler can tell you when they still match, or you made an incompatible change to the code.

This seems like a situation where defining the data structure in two different ways might be good?

Re: Durable Coroutines for Go

#15

Post by deadfa11 »

tedunangst wrote: post #2

This seems really brittle. https://github.com/stealthrocket/coroutine/blob/main/getg_am...

Eh, this one seems a reasonable trade-off to me at this point. If you try to handle every potential issue upfront, you'll never release, and this seems entirely fixable down the road. Persistent coroutines is a pretty challenging area to explore (tried this with Lua in a previous venture), and this seems a pretty minor point in that overall complexity. I'm curious, what else have you come across that justifies the "really brittle" conclusion?

Re: Durable Coroutines for Go

#16

Post by chris6f »

skybrian wrote: post #9

The saved coroutine state is an implicitly-defined data structure and they don’t support any kind of migration when the code changes, so the durability will be quite limited.

We have a solution already, and we're exploring a few more.

The durable coroutine library is one part of a larger system we're releasing soon. See https://stealthrocket.tech/blog/fairy-tales-of-workflow-orch... for more information :)

Re: Durable Coroutines for Go

#17

Post by chris6f »

tedunangst wrote: post #2

This seems really brittle. https://github.com/stealthrocket/coroutine/blob/main/getg_am...

A source-to-source compiler, and a library that bundles runtime implementation details, was the path of least resistance. We'd love to integrate this with the Go compiler (`go build -durable`, vs. `coroc && go build -tags durable`), but the compiler is closed and maintaining a fork of Go is not feasible for us at this time.

Re: Durable Coroutines for Go

#18

Post by ramenmeal »

lelandbatey wrote: post #6
infogulch wrote: post #3

Why are channels insufficient for this use case?

A goroutine-and-channel implementation may not work as well for the "durable" goal, while this implementation is focused on being able to perfectly serialize the state of the Coroutines in order to resume them elsewhere (durable).

Honest question, why not use another technology to store the state between the two? Like a DB or queuing mechanism.

Re: Durable Coroutines for Go

#19

Post by chris6f »

ramenmeal wrote: post #18
lelandbatey wrote: post #6

Earlier quoted context omitted.

A goroutine-and-channel implementation may not work as well for the "durable" goal, while this implementation is focused on being able to perfectly serialize the state of the Coroutines in order to resume them elsewhere (durable).

Honest question, why not use another technology to store the state between the two? Like a DB or queuing mechanism.

The library provides a way to serialize coroutine state, and to later deserialize that state and resume a coroutine from its last yield point. Where you store this state (in a DB, in a queue, etc) is up to you!

Re: Durable Coroutines for Go

#20

Post by infogulch »

ramenmeal wrote: post #18
lelandbatey wrote: post #6

Earlier quoted context omitted.

A goroutine-and-channel implementation may not work as well for the "durable" goal, while this implementation is focused on being able to perfectly serialize the state of the Coroutines in order to resume them elsewhere (durable).

Honest question, why not use another technology to store the state between the two? Like a DB or queuing mechanism.

I guess the internal state of the coroutine won't be automatically preserved. Say you're emitting numbers from an rng. Theoretically with this you could restart the program while keeping the internal rng state so a restart would continue emitting new values from the same sequence; with channels you could preserve the emitted values, but you couldn't automatically save and restore the rng state so the values would diverge from a continuous run at some point.
Post reply on HN