For those not familiar with workflows as code, a workflow is a method that is executed in a way that can't fail—each step the program takes is persisted, so that if execution is interrupted (the process crashes or machine loses power), execution will be continued on a new machine, from the same step, with all local/instance variables, threads, and the call stack intact. It also transparently retries network requests…
Seems like this would depend on some storage guarantees, but I can’t find anything about that
Temporal .NET – Deterministic Workflow Authoring in .NET
31–40 of 51 posts
Re: Temporal .NET – Deterministic Workflow Authoring in .NET
#32For those not familiar with workflows as code, a workflow is a method that is executed in a way that can't fail—each step the program takes is persisted, so that if execution is interrupted (the process crashes or machine loses power), execution will be continued on a new machine, from the same step, with all local/instance variables, threads, and the call stack intact. It also transparently retries network requests…
Sure you can claim this is a fundamentally lower level mechanism, but really it's the same thing.
It is probably important, like all enterprise software quagmires, to consider how they are sold.
Your typical IT manager with low code skills has all his documented high level processes his group manages. Hey look, Visio diagrams with point to point flows of boxes with arrows, and ... maybe ... some interactions of state with databases and/or systems domains.
They know just enough that the actual nuts and bolts is buried in all this ... code. Actual unintelligible code to them, even if they have some inkling of coding.
To them, it's simple flow, and his PHBs above him can understand what he does with the simple flows.
Then some workflow vendor walks in the door, pops up some visual editor, and wows him with basically "you don't need all that code, you put it into the workflow tool and it will just work, and BOOM your coding environment IS your simple visio document".
WOW SIGN ME UP TAKE $$$$$$$! Then comes the pilot flows.
Error handling and retries?
Distributed State and even worse, Transactional update of Distributed state?
Load distribution?
Branching, looping?
Systems integration?
It goes back to the theory of computation. State machines have limitations in processing. The stack machine addresses some of that, but it eventually runs out, requiring ... the turing machine.
As it turns out, almost all enterprise data flows or processes require turing machines. That's why they are coded at some level by turning complete languages.
Superficially at a high level, you start to see a basic state machine model on top of that ... but it is an illusion.
You move the turing machine into the workflow engine (and the workflow engine IS a turing machine ... they all have them: state, looping, branching) and the "simple point to point" flow becomes spaghetti ... tool-locked in spaghetti, with fixed limits on ability to do things.
The current evolution to workflows is the "directed acyclic graph" workflow engine. This has been an improvement, mostly by constraining the actual use of workflow engines to task organizations that they can do, and trying to keep people from going "full Turing" in the workflow engine.
It still can loop ... most do it by recursive calls to subflows ... gets pretty spaghetti. And you still have the fundamental issue that all PHBs will want in the workflow. On error, retry, or have a recovery flow, or that type. Still a huge amount of complexity to properly get the workflow working.
And yet the visual editing workflow tool can have enormous value. Enormous. The Visual nature of the flow, ability to visually diagnose suspended / failed executions. And workflow are everywhere: batch processes, code builds, deployments, automated maintenance, backups / restores, etc.
And I haven't even gotten into the mess of automated rules-engine-based stuff.
The only value structuring low level code along the lines of what "enterprise workflow" has evolved into after decades (useful, but not a holy grail) is if it gives you a fundamentally better way to visualize the execution of the code, which can happen under constrained use of workflow engines.
UML was a massive disaster, another tangential relation to what appears to be being done here. There your "workflows" or code diagrams were code generated to code.
Alas, the final problem of workflow engines is their balkanization. XML standardizations (BPEL) failed miserably for all the usual corporate product standardizations (functioned as lockin for the existing players, lowest common denominator abilities, ugly, XKCD protocol+1).
If only... if only there was a good designed representation scheme and a wide variety of good open source visualization and execution engines. But there aren't.
I think what is discussed here is a step towards a potential solution: it comes from the IDE tooling, something that a workflow always was (in the vein of the now-defunct CASE/Computer Aided Software Engineering days). A standard tooling that coders demand and IDEs provide as a minimum barrier. But IDEs are single machine things, and workflows are distributed entities ... sigh, nevermind that thought.
Ok, maybe we just need a good visualization tool first that is more universal. Don't care about the creation, just something that can "plug in" and represent non-workflow system interactions AS workflows. "Enterprise execution visualization". A REALLY good system for that has never existed IMO, and is universally needed.
Re: Temporal .NET – Deterministic Workflow Authoring in .NET
#33For those not familiar with workflows as code, a workflow is a method that is executed in a way that can't fail—each step the program takes is persisted, so that if execution is interrupted (the process crashes or machine loses power), execution will be continued on a new machine, from the same step, with all local/instance variables, threads, and the call stack intact. It also transparently retries network requests…
Workflows, that is generally speaking the "boxes with arrows between them" pseudo-state machine, really a turing machine, are an interesting thing historically in the "enterprise computing" realm. Sure you can claim this is a fundamentally lower level mechanism, but really it's the same thing. It is probably important, like all enterprise software quagmires, to consider how they are sold. Your typical IT manager with…
> The Visual nature of the flow, ability to visually diagnose suspended / failed executions.
Temporal has a web UI in which you can see which executions are failing, and see on which step they're failing:
Re: Temporal .NET – Deterministic Workflow Authoring in .NET
#34It's for writing code that has steps that involve humans. E.g.: sending a mail notification for someone to approve a document, that kind of thing.
These are inherently slow and asynchronous, because humans operate on timescales of hours or days, not milliseconds.
Re: Temporal .NET – Deterministic Workflow Authoring in .NET
#35For those people commenting here that don't seem to quite understand the point of a workflow library: It's for writing code that has steps that involve humans. E.g.: sending a mail notification for someone to approve a document, that kind of thing. These are inherently slow and asynchronous, because humans operate on timescales of hours or days, not milliseconds.
Temporal, while it uses the "workflow" terminology, is a new type of thing. At a basic level, it's "do you want your backend code to run reliably?" If yes, and you're okay with the latency hit of each step getting persisted for you, then the answer is "use Temporal to write your backend code." It's a new programming model that lets you develop at a higher level of abstraction, where you don't have to be concerned about faults in the hardware or network or downstream services/3rd party APIs being temporarily down. Where you no longer have to code retries, timeouts, use task queues, or use a message bus to communicate between services. And oftentimes don't even need to use a database.
Re: Temporal .NET – Deterministic Workflow Authoring in .NET
#36Earlier quoted context omitted.
Hrmm, I will investigate using an expression tree for this (now's the time while it is alpha). I was hoping to avoid people having to create lambdas. I hope I don't run into overload ambiguity with the existing `ExecuteActivity` calls where you can just pass an existing method as Func param. I will investigate this approach, thanks! Of course the "Ref" pattern is user-choice/suggested-pattern, it's not a requirement…
As another point of reference, Hangfire also uses expressions for a similar use-case and it seems to work quite well. E.g. https://docs.hangfire.io/en/latest/background-methods/passin... BackgroundJob.Enqueue (x => x.Send(13, "Hello!"));
Re: Temporal .NET – Deterministic Workflow Authoring in .NET
#37For those not familiar with workflows as code, a workflow is a method that is executed in a way that can't fail—each step the program takes is persisted, so that if execution is interrupted (the process crashes or machine loses power), execution will be continued on a new machine, from the same step, with all local/instance variables, threads, and the call stack intact. It also transparently retries network requests…
Workflows, that is generally speaking the "boxes with arrows between them" pseudo-state machine, really a turing machine, are an interesting thing historically in the "enterprise computing" realm. Sure you can claim this is a fundamentally lower level mechanism, but really it's the same thing. It is probably important, like all enterprise software quagmires, to consider how they are sold. Your typical IT manager with…
I think they also have C# SDK which I can't vouch for because I haven't used it.
Re: Temporal .NET – Deterministic Workflow Authoring in .NET
#38There’s a lot to learn with it. I’ve seen ramp-up take a few months per engineer, though we’re also making it a little harder on ourselves by self-hosting, being the early adopters on the Python SDK (Go, Java, and TypeScript are the most mature I think), and dealing with a mix of Python async and multiprocessing (a bunch of CPU bound activities in the mix). The docs are solid, and the team is responsive to community users.
Re: Temporal .NET – Deterministic Workflow Authoring in .NET
#39Glad to see that Temporal follows a similar approach and gets you the same benefits. For every coder out there currently using AWS SWF: if your day work involves more than just handholding a handful workflows but building those, take a look at Temporal. You'll never look back.
(To be fair I am still grumpy that they still separate "deciders" and activitities, but I can see the benefits of that.)
If you want to use a GUI to design workflows: equally useful, but probably with a different target audience.