Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The BEAM Has Spoiled Me

gvaughn.github.io

Post reply on HN
Print view View original

41–50 of 88 posts

Re: The BEAM Has Spoiled Me

#41

Post by dmitriid »

ridiculous_fish wrote: post #19

How does BEAM handle system calls? If I have a million BEAM processes and each one calls stat(), what happens?

(has been a while since I touched Erlang, so: grain of salt)

There are several tricks that BEAM employs to work around these problems:

- BEAM has dedicated threads to I/O, and system calls will happen on those dedicated threads. So, the green thread aka processes wil not be affected

- (almost [1]) every call that happens "outside erlang" (that is it calls some code implemented in C inside the VM such as regexps etc.) is re-entrant. So the VM can and will re-prioritize tasks and put processes to sleep when needed and will re-start work when the process wakes up or some external work is done and the answer is received back

So in theory all those stat() calls will be scheduled and queued on the separate prioritized I/O thread, the processes will be put to sleep until the result comes back, and then they will be awoken in turn. This may cause problems with the host OS though :) [2]

[1] There are definitely places where code is not re-entrant yet, because you see updates in release notes from time to time, but most code is re-entrant because of reduction counting: https://news.ycombinator.com/item?id=14440205 and https://stackoverflow.com/questions/31751766/reductions-in-t... and because schedulers can steal processes from each other: https://hamidreza-s.github.io/erlang/scheduling/real-time/pr...

[2] A slightly unrelated anecdote: At a previous job due to some improper coding the web server would slowly accumulate up to to a few 100s of GBs of data in memory due to some long-running processes. When the processes were done, the GC would kick in and release that chunk of memory back to the OS. The OS had trouble with quickly freeing and reclaiming that memory. BEAM was meanwhile happily chugging along as if nothing has happened :D

Re: The BEAM Has Spoiled Me

#42

Post by dmitriid »

sergiotapia wrote: post #26

Been programming in Elixir for years now, I'd say 19 out of 20 engineers I've spoken to don't really leverage BEAM. There's a significant gap between "I write Elixir" and "I leverage BEAM". I'm in the former camp _still_. I just can't find the time/opportunity to dive in and use it to my advantage. Testament to how performant and easy it is to get things up and running in Phoenix I guess. I believe once this secret s…

> I'd say 19 out of 20 engineers I've spoken to don't really leverage BEAM.

That's because most tasks we as engineers write rarely have the need for distribution or parallelism. It's mostly "get data A, get data from B, mix, save resulting data C". nd when you need to do that in parallel, you through it in Kubernetes, or Apache Beam.

It's even more pronounced in web/microservices situations because the web server takes care of all of that for you.

Re: The BEAM Has Spoiled Me

#43

Post by arvinsim »

hazz99 wrote: post #5
grantjpowell wrote: post #2

The BEAM is great, but the Elixir language itself is really what spoiled me. After having written ruby for years, I really enjoy what the Elixir language has to offer. Pattern matching[0] and the pipe operator[1] eliminate a ton of intermediate variables. Functional programming took a while to get used to, but I'm really starting to like thinking immutably. Now that I'm more advanced in the language, I'm starting to…

I love Erlang/Elixir and want to use it more, but I just don’t have any good use cases for it! I don’t often make big concurrent back ends - maybe REST APIs, but that didn’t seem easy last time I tried.

For me it’s job opportunities. In the USA, I think there seems to be enough jobs but globally, it is still sparse.

Re: The BEAM Has Spoiled Me

#44

Post by dynamite-ready »

I've written a decent amount of Elixir for a personal project, and have come to a few conclusions.

For a start, it's very different. That will become self evident if you try to learn either Erlang or Elixir, when coming from a language like Python/C++/etc (and though Elixir looks a lot like Ruby, it's still more like Erlang, tbh). So that can be classed as a negative.

Secondly, fault tolerance in Erlang/Elixir/OTP is not quite as simple as commonly billed. Isolated processes and supervision trees can give you a lot of mileage, but the model is not a silver bullet. I found I still needed to create an error boundary to make an application 'Do the right thing' in very specific error conditions (for example, not killing a running process, when making a single, accidental RPC with the wrong arguments).

Apart from that though, I can't think of much else negative to say about Elixir.

The tools for introspection are amazing. I recently debugged a production issue by opening a console connected to a running system, and calling the functions I suspected to be failing to see what they were returning directly. Not sure if any other language has that facility.

Then there's all the stuff built into Elixir, like the test suite, the build tool, and even an official linter / code formatter.

Erlang interoperability is also a brilliant design decision. Anytime I need to use a library for a task, I look through the Elixir community first. But if I can't find something suitable, there's often Erlang code out there with years of Dev time behind it.

This is just a mere fraction of all I want to say about the language. And most it is very positive.

Re: The BEAM Has Spoiled Me

#45

Post by bigbassroller »

AlchemistCamp wrote: post #12
jefflombardjr wrote: post #3

Earlier quoted context omitted.

Can you recommend any learning materials that were particularly useful to you?

For me personally, Programming Elixir and Programming Phoenix , from the Pragmatic Bookshelf, were both very useful. Quite a few of my students have said great things about Phoenix in Action , from Manning, and it looks to me like a fantastic resource for beginners. Elixir in Action is also top-notch and will show you what's special about the BEAM.

To those out there learning from “Programming Phoenix”, although is the best resource for lay of the land of Phoenix, the content is getting out dated with the rise of auth generator and the context model has also changed a lil bit. It takes diligence in following along and referencing the documentation to not get stuck.

Great book though.

Re: The BEAM Has Spoiled Me

#46

Post by princevegeta89 »

I still write Elixir only, and don't really leverage BEAM (not yet) but the sheer performance of the system amazes me. Phoenix is very largely scalable, and makes some really clever decisions in terms of architecture unlike Rails/Django.

The learning curve has been a little steep and there is a paradigm shift but I've encountered situations where it really ended up helping me navigate through code with precision and clarity. I think Phoenix is the best candidate for a Web Framework as of today, if one desires rapid development. The standard library is rich, documentation is solid, and features like async tests etc are an added bonus.

I'm using Phoenix for our side project and it is swift as a rocket.

Re: The BEAM Has Spoiled Me

#47

Post by lawn »

hazz99 wrote: post #32
OkayPhysicist wrote: post #31

Earlier quoted context omitted.

Phoenix takes a little bit of reading to figure out (though the documentation is brilliant, and really easy to read). Once you understand how it works, though, creating an API is extremely straightforward. It's amazing.

I just couldn’t get past ecto! It never seemed to “click” for me. Maybe I need to go back and learn it more in-depth, it sounds like it’ll be fantastic once I get the hang of it.

I can recommend the book "Programming Ecto". Previously I had only dabbled with ecto in the context of Phoenix, but it didn't really click for me either, but the book does a good job going through the different parts of Ecto and clearing up some of my confusion with it.

Re: The BEAM Has Spoiled Me

#48

Post by tluyben2 »

busterarm wrote: post #24
siscia wrote: post #23

I use to make the same mistake of the author. It is not the BEAM, it is the architecture that the BEAM force upon you to spoils it (the author). There are deeper points in what the author says. Even if you let a process crash in the BEAM, but you don't leave the system in a consistent, recoverable state, the BEAM is not going to help you. Once you realize this, and have a way to apply back pressure, you can get a lot…

The problem there is that your coworkers working in those languages who don't have Erlang/Elixir experience have not internalized those lessons yet and they're not going to understand why you're doing what you're doing when they review your code. Edit: I remember an interview that I had once with a team that was trying to bring on a senior to improve their PHP code and they were really unused to the code style I used…

This is a problem with 'theoretical' (books/uni 'experience') and 'IBM thinking' (people are more expensive than adding more machines, so always optimize for using less people time; just add machines... says the hardware (and per core software license) vendor. Amazon AWS agrees!) people is that they lack the real life experience where things do not fit in 'best practices' (as in design patterns) or green field architectures from books/tutorials if you don't want to scale up to 1000 nodes for 100.000 users. Optimizing this (so you can run those 100.000 on 1 node, but let's say 2 for failover) might not result in code which people might find 'understandable' from their point of view. A lot of people with even years of experience, work in a kind of 'process' way; stacking up code the way they have learned and handle every case as the same case with 'somewhat different logic'. They might not understand your code or they might understand it but think it's not adhering to proper best practices and standards. Their loss.

Re: The BEAM Has Spoiled Me

#49

Post by gotts »

dynamite-ready wrote: post #44

I've written a decent amount of Elixir for a personal project, and have come to a few conclusions. For a start, it's very different. That will become self evident if you try to learn either Erlang or Elixir, when coming from a language like Python/C++/etc (and though Elixir looks a lot like Ruby, it's still more like Erlang, tbh). So that can be classed as a negative. Secondly, fault tolerance in Erlang/Elixir/OTP is…

>I recently debugged a production issue by opening a console connected to a running system, and calling the functions I suspected to be failing to see what they were returning directly. Not sure if any other language has that facility.

Clojure has it.

Re: The BEAM Has Spoiled Me

#50

Post by tluyben2 »

dmitriid wrote: post #42
sergiotapia wrote: post #26

Been programming in Elixir for years now, I'd say 19 out of 20 engineers I've spoken to don't really leverage BEAM. There's a significant gap between "I write Elixir" and "I leverage BEAM". I'm in the former camp _still_. I just can't find the time/opportunity to dive in and use it to my advantage. Testament to how performant and easy it is to get things up and running in Phoenix I guess. I believe once this secret s…

> I'd say 19 out of 20 engineers I've spoken to don't really leverage BEAM. That's because most tasks we as engineers write rarely have the need for distribution or parallelism. It's mostly "get data A, get data from B, mix, save resulting data C". nd when you need to do that in parallel, you through it in Kubernetes, or Apache Beam. It's even more pronounced in web/microservices situations because the web server tak…

Apache Beam I see, but what has Kubernetes to do with getting and mixing data in parallel?
Post reply on HN