Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Discord Scaled Elixir to 5M Concurrent Users

blog.discordapp.com

Post reply on HN
Print view View original

61–70 of 260 posts

Re: How Discord Scaled Elixir to 5M Concurrent Users

#61

Post by janekm »

quaunaut wrote: post #52
janekm wrote: post #46

Earlier quoted context omitted.

I read it a little differently. The whole article is about how Erlang/Elixir fails at its core reason for existence (fast message passing between distributed processes) and all the complicated work-arounds they had to implement to avoid actually using this core feature of Erlang.

I don't think that's what Erlang was made for at all? Erlang was never about speed, it's about reliability, availability, and ease of concurrency. And Elixir just makes it easier to access those tools.

It may well be a misconception on my part - I assumed that as such message passing is such a core feature of using Erlang for distributed systems one should be able to treat it as a low-cost operation.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#62

Post by pencilhappen »

Cieplak wrote: post #20

I know that the JVM is a modern marvel of software engineering, so I'm always surprised when my Erlang apps consume less than 10MB of RAM, start up nearly instantaneously, respond to HTTP requests in less than 10ms and run forever, while my Java apps take 2 minutes to start up, have several hundred millisecond HTTP response latency and horde memory. Granted, it's more an issue with Spring than with Java, and Parallel…

> Parallel Universe's Quasar is basically OTP for Java, so I know logically that Java is basically a superset of Erlang at this point

It's not really a superset. The JVM has a single global heap, whereas BEAM has per-process heaps. Global state is the enemy of concurrency, and the best GC algorithm is one you don't need to run at all. The "several hundred millisecond HTTP response latency and horde (sic) memory" problems you note are in part a result of having a global heap.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#63

Post by [deleted] »

iagooar wrote: post #19

This writeup make me even more convinced of Elixir becoming one of the large players when it comes to hugely scaling applications. If there is one thing I truly love about Elixir, it is the easiness of getting started, while standing on the shoulders of a giant that is the Erlang VM. You can start by building a simple, not very demanding application with it, yet once you hit a large scale, there is plenty of battle-p…

[deleted]

Re: How Discord Scaled Elixir to 5M Concurrent Users

#64

Post by jhgg »

jmcgough wrote: post #47
bsder wrote: post #38

Earlier quoted context omitted.

> I see a bright future for Elixir, and a breath of fresh air for Erlang. Is Erlang really that much of a barrier? Erlang was a touch odd, but I didn't find the language itself that mind-bending. Wrapping my head around the proper way to structure things and the proper use of the OTP libraries was much more time-consuming. That level of architectural thought doesn't magically go away because you changed the language.…

It became open source in the 90s. It's been used by several high-traffic sites/apps to support chat clients (Riot Games, WhatsApp, and I think Fetlife), but outside of that has seen very little adoption. I think it's a lot harder to sell people on a language that looks syntactically unintelligible to a lot of web developers. Most people I know working with Elixir now are/were Rubyists, so it's easy for them to make t…

Elixir/Erlang is great, but in addition to knowing the language, there is a huge library of tools within OTP/ERTS that you need to understand to effectively use it and skills learned from other languages don't map too well to Erlang/Elixir as they would to other languages (like going from Python to Go for example). In addition to learning OTP, you also have to learn how a functional language works if you're not familiar. A lot of things to learn to adopt Elixir/Erlang, but it's well worth it! I just don't think there's much reason for people to make that leap as there are much more familiar languages that can be used to solve the problems most developers are trying to solve.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#66

Post by geogriffin »

janekm wrote: post #46
iagooar wrote: post #19

This writeup make me even more convinced of Elixir becoming one of the large players when it comes to hugely scaling applications. If there is one thing I truly love about Elixir, it is the easiness of getting started, while standing on the shoulders of a giant that is the Erlang VM. You can start by building a simple, not very demanding application with it, yet once you hit a large scale, there is plenty of battle-p…

I read it a little differently. The whole article is about how Erlang/Elixir fails at its core reason for existence (fast message passing between distributed processes) and all the complicated work-arounds they had to implement to avoid actually using this core feature of Erlang.

I'm an erlang fan but I wouldn't claim message passing is "fast" in the sense that you might be thinking of (though copying data does have some GC and CPU cache performance benefits that are harder to reason about.) There's no magic in Erlang; copying data is copying data. The benefits of Erlang lie elsewhere, and I've heard Joe Armstrong, when asked about BEAM performance, say something along the lines of "why do you care about performance [in this day and age]?"

Re: How Discord Scaled Elixir to 5M Concurrent Users

#67

Post by josevalim »

janekm wrote: post #46
iagooar wrote: post #19

This writeup make me even more convinced of Elixir becoming one of the large players when it comes to hugely scaling applications. If there is one thing I truly love about Elixir, it is the easiness of getting started, while standing on the shoulders of a giant that is the Erlang VM. You can start by building a simple, not very demanding application with it, yet once you hit a large scale, there is plenty of battle-p…

I read it a little differently. The whole article is about how Erlang/Elixir fails at its core reason for existence (fast message passing between distributed processes) and all the complicated work-arounds they had to implement to avoid actually using this core feature of Erlang.

People tend to forget that scalability is not a binary property. You always scale up to some users, up to some architecture, up to some amount of nodes. There is no system that will scale to infinity without requiring developer intervention once business needs and application patterns start to settle in.

Distributed Erlang/Elixir has known limitations. For example, the network is fully meshed, which gives you about 60 to 200 nodes in a cluster. Or don't send large data over distribution, as that delays the other messages, etc. Some of those are easily solvable. For example, you can rely on your orchestration tools to break your clusters in groups. Or you can setup an out of band tcp/udp socket for large data. Others may be more complex.

The important question, however, is how far you can go without having to tweak, and, once you reach those roadblocks, how well you can address them. In many platforms, writing a distributed system is a no-no or, at best, they require you to assemble and tweak from day one. In this case, the ability to start with Erlang/Elixir and tweak as you grow is a feature.

And if you never run into those roadblocks, then you can happily continue running on the default stack. Just look at the many companies using Phoenix PubSub and Phoenix Presence, both distributed, without having to worry about fine-tuning the distribution.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#68

Post by YorickPeterse »

didibus wrote: post #60

So, at this point, every language was scaled to very high concurrent loads. What does that tell us? Sounds to me like languages don't matter for scale. In fact, that makes sense, scale is all about parallel processes, horizontally distributing work can be achieved in all language. Scale is not like perforance, where if you need it, you are restricted to a few languages only. That's why I'd like to hear more about pro…

While you don't need language X to scale, certain languages can definitely make it _easier_ and more cost-effective to scale. So it can matter depending on what you're trying to achieve.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#69

Post by quaunaut »

janekm wrote: post #61
quaunaut wrote: post #52

Earlier quoted context omitted.

I don't think that's what Erlang was made for at all? Erlang was never about speed, it's about reliability, availability, and ease of concurrency. And Elixir just makes it easier to access those tools.

It may well be a misconception on my part - I assumed that as such message passing is such a core feature of using Erlang for distributed systems one should be able to treat it as a low-cost operation.

You can generally, just Discord is getting to a level of scale where the issue has more to do with architecture than language. All of their solutions in this post were Elixir-based solutions, and have very clean, easy APIs(and so far not very much code- all those libraries that solved their problems came out to ~400 LOC).

I guess the message here is, "There's never a magic bullet, but you've got everything you need to make yours."

Re: How Discord Scaled Elixir to 5M Concurrent Users

#70

Post by nateberkopec »

didibus wrote: post #60

So, at this point, every language was scaled to very high concurrent loads. What does that tell us? Sounds to me like languages don't matter for scale. In fact, that makes sense, scale is all about parallel processes, horizontally distributing work can be achieved in all language. Scale is not like perforance, where if you need it, you are restricted to a few languages only. That's why I'd like to hear more about pro…

If you have a share-nothing architecture, then yes, any language can scale to any load, some with more hardware than others.
Post reply on HN