Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building Reliable Distributed Systems in Node.js

temporal.io

Post reply on HN
Print view View original

11–18 of 18 posts

Re: Building Reliable Distributed Systems in Node.js

#11

Post by mbj111 »

MuffinFlavored wrote: post #10
hot_gril wrote: post #5

Earlier quoted context omitted.

Sounds like the whole point is you don't have to take anything special into account. Edit: Except determinism?

https://github.com/temporalio/hello-world-project-template-j... You have to write your code using Temporal SDK. At a quick glance: main() calls WorkflowServiceStubs/WorkflowClient I also see something called an "Activity" Also see something called a Worker. Not trying to argue. Genuinely curious if you think this is within the realm of "not take anything special into account" (be forced to use a specific SDK and lay…

100% like it or not vendor lock-in is there with all workflow solutions. We need standardisation with workflow solutions as mentioned here in detail https://twitter.com/gwenshap/status/1505950830767206400?s=46...

Re: Building Reliable Distributed Systems in Node.js

#12

Post by lorendsr »

hot_gril wrote: post #6

Never heard of durable execution until now, but I've wondered about it. When I write backend code, I have to keep asking myself "what happens if the server goes down during this line of code?" This is often an issue in the middle of a customer order, like the example here. I end up relying on the database for very many tiny little things, like recording the fact that the user initiated an order before I start to proc…

Yeah, I’ve also written this write-to-db-after-each-meaningful-line-of-code style code, and this is a great improvement. See the first 20m of this talk for an example: https://youtu.be/EFIF8gk9zy8

Starting a workflow is currently ~40ms, and I think we’ll be able to get down to 10ms this year. How long it takes to complete depends on how many persisted steps it takes (and whether it has to wait on an external event). The only steps that are persisted are workflow api calls like sleep(), startChildWorkflow(), or calling code that might fail (ie “Activity”, like a network request).

Re: Building Reliable Distributed Systems in Node.js

#13

Post by lorendsr »

lakomen wrote: post #8

Or you could not use a scripting language and save 50 times the cost

We have Go and Java SDKs that have better performance characteristics if that’s what you’re optimizing for. I think for many businesses, optimizing for development speed is a higher priority (eg if the devs already know JS, use that). The Node runtime with v8 isolates is also able to better protect developers from writing non deterministic code (durable code must be deterministic). More info on that: https://temporal.io/blog/intro-to-isolated-vm

Re: Building Reliable Distributed Systems in Node.js

#14

Post by lorendsr »

hot_gril wrote: post #5
MuffinFlavored wrote: post #3

Earlier quoted context omitted.

you have to code your entire architecture around this premise though, no? aka start from scratch and write things a certain way

Sounds like the whole point is you don't have to take anything special into account. Edit: Except determinism?

The point is that you can write code instead of JSON/YAML like traditional microservice orchestration like AWS step functions. And it’s not a limited dsl—you have the full lang at your disposal, with the one requirement that deterministic code (workflows / the durable code) is in separate functions from non deterministic code (like making a network request, called “Activities”).

Re: Building Reliable Distributed Systems in Node.js

#15

Post by hot_gril »

lorendsr wrote: post #12
hot_gril wrote: post #6

Never heard of durable execution until now, but I've wondered about it. When I write backend code, I have to keep asking myself "what happens if the server goes down during this line of code?" This is often an issue in the middle of a customer order, like the example here. I end up relying on the database for very many tiny little things, like recording the fact that the user initiated an order before I start to proc…

Yeah, I’ve also written this write-to-db-after-each-meaningful-line-of-code style code, and this is a great improvement. See the first 20m of this talk for an example: https://youtu.be/EFIF8gk9zy8 Starting a workflow is currently ~40ms, and I think we’ll be able to get down to 10ms this year. How long it takes to complete depends on how many persisted steps it takes (and whether it has to wait on an external event).…

> The only steps that are persisted are workflow api calls like sleep(), startChildWorkflow(), or calling code that might fail (ie “Activity”, like a network request).

Ok, that's what I was wondering. Makes a lot more sense this way.

Re: Building Reliable Distributed Systems in Node.js

#16

Post by hot_gril »

MuffinFlavored wrote: post #10
hot_gril wrote: post #5

Earlier quoted context omitted.

Sounds like the whole point is you don't have to take anything special into account. Edit: Except determinism?

https://github.com/temporalio/hello-world-project-template-j... You have to write your code using Temporal SDK. At a quick glance: main() calls WorkflowServiceStubs/WorkflowClient I also see something called an "Activity" Also see something called a Worker. Not trying to argue. Genuinely curious if you think this is within the realm of "not take anything special into account" (be forced to use a specific SDK and lay…

I thought this comment chain was about durable execution in general. Temporal seems to be that plus some RPC stuff that is a lot more than "nothing special."

Re: Building Reliable Distributed Systems in Node.js

#17

Post by lorendsr »

hot_gril wrote: post #16
MuffinFlavored wrote: post #10

Earlier quoted context omitted.

https://github.com/temporalio/hello-world-project-template-j... You have to write your code using Temporal SDK. At a quick glance: main() calls WorkflowServiceStubs/WorkflowClient I also see something called an "Activity" Also see something called a Worker. Not trying to argue. Genuinely curious if you think this is within the realm of "not take anything special into account" (be forced to use a specific SDK and lay…

I thought this comment chain was about durable execution in general. Temporal seems to be that plus some RPC stuff that is a lot more than "nothing special."

All the durable execution systems have to run your code in certain way that persists steps like RPCs (and need to provide a mechanism for you to tell the system which functions have RPCs) so they can recover in case of process failures. They all also happen to provide common orchestrator features like retries and timeouts because devs find it useful.

Re: Building Reliable Distributed Systems in Node.js

#18

Post by barbarbar »

lorendsr wrote: post #2

This post talks about the durable execution systems, which include Azure Durable Functions, Amazon SWF, Uber Cadence, Infinitic, and Temporal. Durable execution systems run our code in a way that persists each step the code takes. If the process or container running the code dies, the code automatically continues running in another process with all state intact, including call stack and local variables. Durable execu…

Is this similar to apache camel or spring integration?
Post reply on HN