Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Process-Based Concurrency: Why Beam and OTP Keep Being Right

variantsystems.io

Post reply on HN
Print view View original

21–30 of 58 posts

Re: Process-Based Concurrency: Why Beam and OTP Keep Being Right

#21

Post by IsTom »

Twisol wrote: post #4
rapsey wrote: post #3

> Backpressure is built in. If a process receives messages faster than it can handle them, the mailbox grows. This is visible and monitorable. You can inspect any process’s mailbox length, set up alerts, and make architectural decisions about it. Contrast this with thread-based systems where overload manifests as increasing latency, deadlocks, or OOM crashes — symptoms that are harder to diagnose and attribute. Sorry…

Yes, this is missing the "pressure" part of "backpressure", where the recipient is able to signal to the producer that they should slow down or stop producing messages. Observability is useful, sure, but it's not the same as backpressure.

Sending message to a process has a cost (for purposes of preemption) relative to the current size of receiver's mailbox, so the sender will get preempted earlier. This isn't perfect, but it is something.

Re: Process-Based Concurrency: Why Beam and OTP Keep Being Right

#23

Post by [deleted] »

socketcluster wrote: post #6

The Node.js community had figured this out long before BEAM or even Elixir existed. People tried to introduce threads to Node.js but there was push-back for the very reasons mentioned in this article and so we never got threads. The JavaScript languages communities watch, nod, and go back to work.

[deleted]

Re: Process-Based Concurrency: Why Beam and OTP Keep Being Right

#24

Post by emperorz0 »

Zero-sharing message passing is known. But what about shared state? Given the majority of systems manage shared access to arbitrarily constrained shared state or shared resources, I'd be interested to see how this should be handled without just saying "database". Maybe another article?

Re: Process-Based Concurrency: Why Beam and OTP Keep Being Right

#26

Post by BitsAndObjects »

librasteve wrote: post #15
baud9600 wrote: post #13

Earlier quoted context omitted.

Once you were familiar with occam you could see deadlocks in code very quickly. It was a productive way to build scaled concurrent systems. At the time we laughed at the idea of using C for the same task

I spreadsheeted out how many T424 die per Apple M2 (TSMC 3nm process) - that's 400,000 CPUs (about a 600x600 grid) at say 1GIPs each - so 400 PIPS per M2 die size. Thats for 32 bit integer math - Inmos also had a 16 bit datapath, but these days you would probably up the RAM per CPU (8k, 16k?) and stick with 32-bit datapath, but add 8-,16-bit FP support. Happy to help with any VC pitches!

David May and his various PhD students over the years have retried this pitch repeatedly. And Graphcore had a related architecture. Unfortunately, while it’s great in theory, in practice the performance overall is miles off existing systems running existing code. There is no commercially feasible way that we’ve yet found to build a software ecosystem where all-new code has to be written just for this special theoretically-better processor. As a result, the business proposal dies before it even gets off the ground.

(I was one of David’s students; and I’ve founded/run a processor design startup raised £4m in 2023 and went bust last year based on a different idea with a much stronger software story.)

Re: Process-Based Concurrency: Why Beam and OTP Keep Being Right

#27

Post by socketcluster »

leoc wrote: post #7
socketcluster wrote: post #6

The Node.js community had figured this out long before BEAM or even Elixir existed. People tried to introduce threads to Node.js but there was push-back for the very reasons mentioned in this article and so we never got threads. The JavaScript languages communities watch, nod, and go back to work.

You may be thinking of some recent round of publicity for BEAM, but BEAM is a bit older than JavaScript.

Haha. I guess the BEAM people can nod down at me with contempt and I nod down at the Elixir folks.

Re: Process-Based Concurrency: Why Beam and OTP Keep Being Right

#28

Post by rustyhancock »

I love the idea of Erlang (and by association Elixir), OTP, BEAM...

In practice? Urgh.

The live is all so cerebral and theoretical and I'm certain the right people know how to implement it for the right tasks in the right way and it screams along.

But as yet no one has been able to give me an incling of how it would work well for me.

I read learn you some Erlang for great good quite a while back and loved the idea. But it just never comes together for me in practice. Perhaps I'm simply in the wrong domain for it.

What I really needed was a mentor and existing project to contribute to at work. But it's impossible to get hold of either in the areas I'm in.

Re: Process-Based Concurrency: Why Beam and OTP Keep Being Right

#29

Post by lou1306 »

BitsAndObjects wrote: post #25

How closely is BEAM/OTP related to the foundational work on CSP (and the implementation in Occam/Transputer way back when…)?

Good question! It's a bit of a stretch. BEAM has mailboxes, non-blocking sends, and asynchronous handling of messages, whereas the original CSP is based on blocking sends and symmetric channels. Symmetric means you have no real difference between sends and receives: two processes synchrnoise when they are willing to send the same data on the same channel. (A "receive" is just a nondeterministic action where you are willing to send anything on a channel).

Occam added types to channels and distinguished sends/receives, which is the design also inherited by Go.

In principle you can emulate a mailbox/message queue in CSP by a sequence of processes, one per queue slot, but accounting for BEAM's weak-ish ordering guarantees might be complicated (I suppose you should allow queue slots to swap messages under specific conditions).

Re: Process-Based Concurrency: Why Beam and OTP Keep Being Right

#30

Post by sriram_malhar »

emperorz0 wrote: post #24

Zero-sharing message passing is known. But what about shared state? Given the majority of systems manage shared access to arbitrarily constrained shared state or shared resources, I'd be interested to see how this should be handled without just saying "database". Maybe another article?

One process is made the logical goto for all operations on that data. The process identity is logically the identity of the shared state.

In other words, it is exactly a database, albeit an in-memory one.

Post reply on HN