Live data from Hacker News

Elixir saves Pinterest $2M a year in server costs

paraxial.io

381–390 of 481 posts

Re: Elixir saves Pinterest $2M a year in server costs

#381
post #324

Earlier quoted context omitted.

It’s not even remotely similar. Node’s cluster is just bog-standard OS subprocesses running their own event loop. To spread work over multiple cores with the cluster (or even worker-threads) modules you have to do so explicitely and manually. It’s essentially the same model you get with pthreads, or java, or python. BEAM is a completely different model, the “processes” are internal tasks, which the runtime will sched…

I'm unfamiliar with BEAM. How does this compared to goroutines? Obviously they won't migrate between machines, but concurrency feels very easy and ergonomic in Go.

At the most basic level it's a bit similar: there is a multithreaded runtime which schedules work in userland. Green threads if you will.

The devil, however, is once you go beyond the trivial.

First, the units of work operate completely differently, BEAM follows the actor model rather than CSP, meaning every actor has an address / mailbox and the actors can send one another messages through this, any actor can send any other actor (they're aware of) messages, and actors can process their mailboxes however they want.

But BEAM is also completely strict and steadfast about its actor model: its actors are processes, each has its own stack but also its own heap, when one actor sends a message to an other, the message content gets copied (/ moved) from one stack or heap to an other, processes do not share memory[0]. Incidentally this is what makes distribution relatively transparent (not entirely so, but impressively still): if everything you do out to interact with others is send asynchronous one-way messages, it doesn't really matter whether they're in the same process (OS), in a different process (OS), or on a different machine entirely.

The reason BEAM works like that however is not for any sort of theoretical purity, instead it is in service to reliability, which is the second big difference between BEAM and Go: BEAM error handling is inter-process, not intra-process. BEAM's error handling philosophy is that processes encounter errors, for all sort of reasons, and when that happens you can't know the entire state of the process, so you just kill it[1] and it should have one of its buddies which is linked to it, and whose job is to handle this and orchestrate what happens.

BEAM has built-in support for linking and monitoring. In the context of erlang, linking means that if one process dies (crashes), the other is sent a special message which also kills it. This message can be received as a normal message instead, in order to handle the crash of your sibling (in which case you receive various metadata on the crash). Monitoring means you just want to receive the crash signal. The reason you might prefer linking to monitoring is that if you're a manager of other processes and you crash, you probably want all the processes you manage to die as well. Which doesn't happen with monitors.

That is because BEAM has its origins in telecommunications, where reliability means redundancy, and oversight. So the way you structure an application in beam (often) is a tree of processes, where many of the processes have oversight of a subtree, handle fuckups (maybe by restarting, maybe by something else), serve as entry point to their workers, etc..., and if one of the leaves dies that's just a signal sent to its parent, which might just die and signal its parent, which will handle it somehow. This is the design principle known as the supervision tree: https://www.erlang.org/doc/design_principles/des_princ#super...

The third big difference is more philosophical and has to do with code reuse: because of (2) above, a lot of erlang / beam / otp is communicating between processes in a subtree, moving messages between them, exit signal strategies, etc... which leads to behaviours (https://www.erlang.org/doc/design_principles/des_princ#behav...), which are pretty alien because they're more or less mini frameworks, which not only are two things which are usually put opposite one another, but many people don't really want to hear about frameworks.

But that's what they are: behaviours are the encoding of entire prototypal lifecycle and communication patterns, where the user / implementer of the behaviour fills in the "business" bits.

Oh yeah and beam comes with an entire suite of introspectability tooling, which is kinda linked to (2): all the oversight thing ends up at people, so you can connect to a runtime and look at and touch all the things, more or less.

BEAM is a bit of an OS running on an OS really, probably closer in philosophy to the image-based languages of the 80s. In part because it is a language from the 80s. Not quite image based though, or in an other way designed to go even further and just run forever, as it includes built-in support for hot code reloading and migrations (though from what I remember that's not super great or fun, it was quite messy and involved to actually do properly).

By comparison to all that, goroutines are just threads which happen to be cheap so you can have lots.

[0] kinda, some objects live on shared heaps as an optimisation but they're immutable and reference counted so it's an implementation detail.

[1] and here if actors share any memory, an actor might be dying in the middle of updating or holding onto shared state, which means its error corrupts other actors touching the same state

Re: Elixir saves Pinterest $2M a year in server costs

#382

Earlier quoted context omitted.

I don’t tend rewrite Python code any more often than is needed due to feature changes or occasionally refactoring to pay off some maintenance friction caused by design that has in practice turned out to be suboptimal. What are you talking about?

When you move from one minor rev of python to the next, some language feature changes (either syntax or semantics or features no longer work.) For instance... if you use async io in 2.x, the debugger stops working. Between 2.3, 2.5 and 2.7, the syntax of package variable scoping changed and then the semantics changed from package to class variables. If you used a feature like package variables in your code in 2.3, th…

> When you move from one minor rev of python to the next, some language feature changes

Even if there was a breaking change affecting your project every minor version, to have the cadence of backward-compatibility induced changes you siggest you’d have to be switching Python versions forward about four times as fast as they are released, which, if you started with the oldest in support version at the beginning of the project, you could only sustain for about a year and a half, before running out of versions to switch forward to.

> If you used a feature like package variables in your code in 2.3,

Then you are probably out yelling at kids to get off your lawn; 2.3 being out of support for 12 years.

Re: Elixir saves Pinterest $2M a year in server costs

#384
post #64

Earlier quoted context omitted.

> When our notifications system was running on Java, it was on 30 c32.xl instances. When we switched over to Elixir, we could run on 15. Would be curious to know how they tried to optimise the Java stack. Because on every benchmark I've seen the JVM is faster in every which way than Elixir. Except for memory where often people will over-provision the JVM rather than look at where their code might be over-allocating o…

The advantages of Elixir are not performance-related. There is a lot of focus on raw performance on web-related services, when in reality most of their running time is spent waiting for IO. If there are two things the BEAM excels at, is IO and turning almost any problem into half a dozen processes that are scheduled and run in parallel, if not geographically-distributed, with 1/50th the effort of any other language.…

All of what you mentioned applies equally well, if not better, to the JVM. Especially now that it has virtual threads. The article does not go into details about the implementation of the Java program. My guess is that they were not using asyc code, or profiled it to see what the bottlenecks are. Rewriting is always fun especially in the latest flavor of the day stack.

Re: Elixir saves Pinterest $2M a year in server costs

#385

I'm glad to read this and I think these advantages have their place, but... Startups rarely die because of server bill specifically. More often it's lack of PMF, slow iteration, expensive labor. $2M is pocket change for Pinterest, they probably spend more on office snacks, so even if you grow to that scale, the savings in this example are not exactly life-or-death. What is life-or-death, however, is how easy it is to…

The bigger cost is that it takes a lot more devs to do the same thing in Python or especially in Java. Productivity per developer is hands-down Elixir's strongest selling point, not reducing server costs. Python and Java have a multitude of web framework authors and yet none have made anything with the capabilities and ergonomics of Phoenix. I don't think they will either. It would take features those languages lack.

> more devs to do the same thing in Python or especially in Java

i guess it totally depends on "the thing" in question, but do you have any references for that assertion? that common web framework-ish, orm-ish stuff takes a lot more devs in python and especially in java?

Re: Elixir saves Pinterest $2M a year in server costs

#386
post #90

Earlier quoted context omitted.

[flagged]

In what planet? This has absolutely no bearing to the Python experience. And I'm using the stuff for 20 years.

That's great that your 5 line scripts don't use features that change between revs, but people who have to maintain large python apps have to book time to pore over the latest language version's definition, update our linting tools to find where in the codebase we use a deprecated feature, change the code, update the tests, retest and redeploy.

Not to mention getting a version clean dependency closure. Though we have forked and rewritten some of the non-standard modules we're dependent on to be less broken and to give credit where due, it does seem like standard modules supporting python3 are version clean unlike python2 and 1.6.

The heuristic we use is about an hour of dev time per 750 lines of code so our 70,000 line legacy python app takes somewhere around 100 dev hours per minor revision upgrade.

Compare this to a legacy C application written in 1989. How do we port it to the latest version of C? We just copy it and compile. That community went to a lot of trouble to ensure code written in previous versions of the language still worked. The last time I heard of a language feature being deprecated was in 2011 (though I think gcc recently undefeated support for trigraphs.)

In my opinion, your python baby is ugly. It was ugly in 1.6. It was ugly in 2.x. And it remains ugly in the 3.x era. You should come to terms with the fact that some people just don't like python.

Re: Elixir saves Pinterest $2M a year in server costs

#387

Earlier quoted context omitted.

They rewrote, which is known to help too. Going from 30 to 15 instances is not bad but it's very likely that a Java-to-Java rewrite would have helped go down too. The big one however is going from 200 Python servers to 4 Erlang ones: a 50x reduction is quite something and a Python-to-Python rewrite would not have allowed to achieved a 50x gain: > All this is possible because Elixir, and the Erlang platform underneath…

> They rewrote, which is known to help too. Going from 30 to 15 instances is not bad but it's very likely that a Java-to-Java rewrite would have helped go down too. A java-to-java rewrite could have been as much if not more painful than a java-to-elixir rewrite, especially if the service is highly concurrent. Rewriting synchronous code as asynchronous in Java is a lot of work and not fun at all IMO. Given the recent…

Wait a few years for a blog post outlining their rewrite in Java's virtual threads for an improvement in performance and latency ;)

Re: Elixir saves Pinterest $2M a year in server costs

#388

Earlier quoted context omitted.

You can do that easily in modern Java--even for older JVMs, tools like Netty and later Vertx have been around forever. Or in Node, even more easily. Elixir/BEAM do have some benefits that are worth considering for many projects. But they absolutely are not special in this regard, and that's the junior-developer trap about which the person to whom you replied was referring.

I'm a seasoned Java and Node developer, but have never touched Elixir/Erlang. Could you spell out for me the benefits Elixir provides over Java concurrent code? Is it actually a performance gain, or simply a nicer syntax? I am a bit confused by the claims of this post and of the earlier comment. Thanks a lot

So, when you're working with Elixir or any language on the BEAM VM, you're in a world where data is immutable and processes are isolated. It's like having Akka's actor model but at the VM level, so it's super integrated.

The BEAM VM itself is a different beast compared to the JVM. It's more like its own mini-OS designed for real-time multitasking. Each process has its own garbage collection, and it's all non-blocking. So if one process goes belly-up, it doesn't take the whole system with it. Imagine a network of telephone switches; if one gets zapped by lightning, the rest keep chugging along. That's the level of fault tolerance Erlang and BEAM were designed for.

Now, speaking of fault tolerance, let's talk about how easy it is to mess up an Akka system if you're not careful. Say a new Java dev joins your team and doesn't get Akka's actor model. They might introduce shared mutable state between actors, which is a big no-no and can lead to all sorts of race conditions. Or they might do something like put a blocking operation inside an actor, which can hog resources and mess up the whole system's performance. Akka's great, but if you don't follow its principles, you can still shoot yourself in the foot.

So, the beauty of Elixir and BEAM is that a lot of these good practices are enforced by the VM itself. You get that fault tolerance and concurrency baked right in, without having to rely on every engineer knowing all the best practices.

Hope that clears things up!

Re: Elixir saves Pinterest $2M a year in server costs

#389

Earlier quoted context omitted.

I'm a seasoned Java and Node developer, but have never touched Elixir/Erlang. Could you spell out for me the benefits Elixir provides over Java concurrent code? Is it actually a performance gain, or simply a nicer syntax? I am a bit confused by the claims of this post and of the earlier comment. Thanks a lot

So, when you're working with Elixir or any language on the BEAM VM, you're in a world where data is immutable and processes are isolated. It's like having Akka's actor model but at the VM level, so it's super integrated. The BEAM VM itself is a different beast compared to the JVM. It's more like its own mini-OS designed for real-time multitasking. Each process has its own garbage collection, and it's all non-blocking…

That's interesting, thank you! Now I have new things to learn on my list :)

Re: Elixir saves Pinterest $2M a year in server costs

#390
post #349
post #238

Earlier quoted context omitted.

The go runtime has similar capabilities as the BEAM runtime when it comes to concurrent workloads. Go has the benefit of being a typesafe compiled language which gives it speed benefits. But using either one of them instead of Java is probably going to be a huge win for most teams on concurrent workloads.

> The go runtime has similar capabilities as the BEAM runtime when it comes to concurrent workloads. Only if you think that the BEAM is similar to being able to easily spawn a function on a separate thread and having channels. Last I checked, goroutines had no mailboxes, supervisors, process monitoring, registry, and its scheduler has a much smaller scope and featureset than the BEAM's. I swear it's obvious when peop…

Exactly, you've hit the nail on the head. While Go's runtime does offer some nice concurrency features, like goroutines and channels, it's just not on the same level as BEAM when it comes to a comprehensive approach to fault tolerance and system resilience.

BEAM's got this whole ecosystem built around it, right? Mailboxes, supervisors, process monitoring, and a registry—these are all first-class citizens in the Erlang world. And let's not forget the scheduler; it's like comparing a Swiss Army knife to a simple pocket knife when you look at BEAM's scheduler next to Go's.

It's kinda funny when people talk about BEAM and Erlang as if they're just another runtime or language. (it's more OS like than traditional VM like) They're really more like a whole philosophy of how to build robust, fault-tolerant systems. And if you haven't actually built something substantial with it, you're likely to miss out on what makes it so special.

Couldn't agree more with your take.

Post reply on HN