Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Process-Based Concurrency: Why Beam and OTP Keep Being Right

variantsystems.io

Post reply on HN
Print view View original

41–50 of 58 posts

Re: Process-Based Concurrency: Why Beam and OTP Keep Being Right

#41

Post by matthiasl »

rapsey wrote: post #3

> Backpressure is built in. If a process receives messages faster than it can handle them, the mailbox grows. This is visible and monitorable. You can inspect any process’s mailbox length, set up alerts, and make architectural decisions about it. Contrast this with thread-based systems where overload manifests as increasing latency, deadlocks, or OOM crashes — symptoms that are harder to diagnose and attribute. Sorry…

It took me a while to realise that you were responding to the article, not a comment here.

You're right in correcting the article, but I'd like to add that for probably around a decade, Erlang had 'sender punishment', which is what 'IsTom' who replied to you is probably talking about.

Ulf Wiger referred to sender_punishment as "a form of backpressure" (Erlang-questions mailing list, January 2011). 'sender punishment' was removed around 2018, in ad72a944c/OTP14667. I haven't read the whole discussion carefully, but it seems to be roughly "it wasn't clear that sender punishment solved more problems than it caused, and now that most machines are multi-core, that balance is tipped even more in favour of not having 'sender punishment'".

Re: Process-Based Concurrency: Why Beam and OTP Keep Being Right

#42

Post by linkdd »

epicepicurean wrote: post #36

A rewrite of a stateful application written in python with postgres would be more illustrative of how you're solving the same problems but better. Do BEAM applications not use an actual databse? How is crash tolerance guaranteed? In a typical application I'd write crash tolerance would be handled by the DB. So would transactionality. Without it, one would have to persist each message to disk and be forced to make eve…

You would have a process handling the calls to the postgres.

That process has as local state the database connection and receive messages that are translated to SQL queries, here 2 scenarios are possible:

1) The query is invalid (you are trying to inert a row with a missing foreign key, or wrong data type). In that case, you send the error back to the caller.

2) There is a network problem between your application and the database (might be temporary).

You just let the process crash (local state is lost), the supervisor restarts it, the restarted process tries to connect back to the database (new local state). If it still fails it will crash again and the supervisor might decide to notify other parts of the application of the problem. If the network issue was temporary, the restart succeeds.

Before crashing, you notified the caller that there was a problem and he should retry.

Now, for the caller. You could start a transient process in a dynamic supervisor for every query. That would handle the retry mechanism. The "querier process" would quit only on success and send the result back as a message. When receiving an error, it would crash and then be restarted by the supervisor for the retry.

There are plenty of other solutions, and in Elixir you have "ecto" that handles all of this for you. "ecto" is not an ORM, but rather a data-mapper: https://github.com/elixir-ecto/ecto

Re: Process-Based Concurrency: Why Beam and OTP Keep Being Right

#43

Post by Jtsummers »

nnevatie wrote: post #38

BEAM/OTP are great, but do impose an exotic language onto the user. Most programs and solutions of today aren't Erlang-based.

Erlang is a pretty simple language, it's hardly "exotic". Any competent programmer should be able to pick it up in a short period of time, days to weeks. Now, how long to master the concurrency model, "let it crash" mindset, small processes, and supervisors? Maybe a bit longer.

Re: Process-Based Concurrency: Why Beam and OTP Keep Being Right

#45

Post by nnevatie »

Jtsummers wrote: post #43
nnevatie wrote: post #38

BEAM/OTP are great, but do impose an exotic language onto the user. Most programs and solutions of today aren't Erlang-based.

Erlang is a pretty simple language, it's hardly "exotic". Any competent programmer should be able to pick it up in a short period of time, days to weeks. Now, how long to master the concurrency model, "let it crash" mindset, small processes, and supervisors? Maybe a bit longer.

Yeah, I meant this from the perspective that for example, I'd love to have that environment and approach in my sleeve, but utilizing C++ and all I have built using it.

Re: Process-Based Concurrency: Why Beam and OTP Keep Being Right

#46

Post by Jtsummers »

dzonga wrote: post #40

inverse thinking is needed here - instead of having a solution trying to find a problem. what would it look like if you didn't need concurrency at all - would simply having a step by step process enough e.g using DAGs what would it look like if by not letting it crash - you can simply redo the process like a Traditional RDBMS does i.e ACID they're domains where OTP / BEAM are useful - but for majority of business cas…

> what would it look like if you didn't need concurrency at all - would simply having a step by step process enough e.g using DAGs

What business systems don't use concurrency in some form? I can only think of the simplest data processing tasks written for batch processing. But even every embedded system I've ever developed or worked on used concurrency. Though for older systems this was often hand rolled, and as error prone as you might expect. For newer systems (developed this century), it was often done using a task system baked into the embedded RTOS.

Re: Process-Based Concurrency: Why Beam and OTP Keep Being Right

#47

Post by vipulbhj »

rustyhancock wrote: post #28

I love the idea of Erlang (and by association Elixir), OTP, BEAM... In practice? Urgh. The live is all so cerebral and theoretical and I'm certain the right people know how to implement it for the right tasks in the right way and it screams along. But as yet no one has been able to give me an incling of how it would work well for me. I read learn you some Erlang for great good quite a while back and loved the idea. B…

I personally really really enjoy writing Elixir. It is a really intuitive way to write programs. Phoenix is a great web framework, and I think all of it is quite approachable. We just had a go programmer start at our org recently and they were contributing to one of our Phoenix bases SaaS apps within weeks

Re: Process-Based Concurrency: Why Beam and OTP Keep Being Right

#48

Post by vipulbhj »

josefrichter wrote: post #39
nnevatie wrote: post #38

BEAM/OTP are great, but do impose an exotic language onto the user. Most programs and solutions of today aren't Erlang-based.

Use Elixir. Not exotic at all.

And if you don't like that, you could try some other BEAM language https://github.com/stars/michallepicki/lists/beam-languages

Re: Process-Based Concurrency: Why Beam and OTP Keep Being Right

#49

Post by rustyhancock »

vipulbhj wrote: post #47
rustyhancock wrote: post #28

I love the idea of Erlang (and by association Elixir), OTP, BEAM... In practice? Urgh. The live is all so cerebral and theoretical and I'm certain the right people know how to implement it for the right tasks in the right way and it screams along. But as yet no one has been able to give me an incling of how it would work well for me. I read learn you some Erlang for great good quite a while back and loved the idea. B…

I personally really really enjoy writing Elixir. It is a really intuitive way to write programs. Phoenix is a great web framework, and I think all of it is quite approachable. We just had a go programmer start at our org recently and they were contributing to one of our Phoenix bases SaaS apps within weeks

It's the converse that's an issue. If your org doesn't use any Erlang.

You're not going to be able to add it.

I don't find that to be true of many other ecosystems.

We could and do have a few Rust tools and webapps.

There is a few older Python/Flask internal applications.

If I went to an org with established tools from the ecosystem then that is not a problem!

Re: Process-Based Concurrency: Why Beam and OTP Keep Being Right

#50

Post by OkayPhysicist »

dzonga wrote: post #40

inverse thinking is needed here - instead of having a solution trying to find a problem. what would it look like if you didn't need concurrency at all - would simply having a step by step process enough e.g using DAGs what would it look like if by not letting it crash - you can simply redo the process like a Traditional RDBMS does i.e ACID they're domains where OTP / BEAM are useful - but for majority of business cas…

If you don't need concurrency, then you simply don't need to define any concurrency segmentation. But the real world is wildly concurrent, and most programs will eventually benefit from some degree of concurrency (especially when you can leverage that concurrency into parallelism), so it's beneficial to work in an environment where that improvement can be incremental rather than "we need do a complete rearchitecture to support n=2".

"letting it crash" in BEAM terms often means "simply redo the process". The difference is you end up defining your "transaction" (to borrow database terminology) by concurrency lines. What makes it so pleasant in practice is that you take a bunch of potential failure modes and lump them into a single, unified "this task cannot be completed" failure mode, which includes ~impossible to anticipate failure states, and then only have to expressly deal with the failure modes that do have meaningful resolutions within a task.

With that understanding in mind, I'd argue that nearly all business cases benefit from the BEAM. It's mostly one-off scripts and throwaway tools that don't.

Post reply on HN