Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Discord Scaled Elixir to 5M Concurrent Users

blog.discordapp.com

Post reply on HN
Print view View original

81–90 of 260 posts

Re: How Discord Scaled Elixir to 5M Concurrent Users

#81

Post by pmarreck »

quaunaut wrote: post #71
jhgg wrote: post #64

Earlier quoted context omitted.

Elixir/Erlang is great, but in addition to knowing the language, there is a huge library of tools within OTP/ERTS that you need to understand to effectively use it and skills learned from other languages don't map too well to Erlang/Elixir as they would to other languages (like going from Python to Go for example). In addition to learning OTP, you also have to learn how a functional language works if you're not famil…

I mean, personally I'm using Elixir more for the language itself than OTP. I have a basic knowledge of OTP now after using the language for 2 years, but FP was my main reason for coming and staying, which for me has been the real fun of it. Then again, FP for me was not difficult at all, and actually made several of the issues I had with my Ruby code not being expressive/obvious enough, or difficulty in composition,…

> and actually made several of the issues I had with my Ruby code not being expressive/obvious enough, or difficulty in composition, disappear overnight.

I've noticed that pattern-matching and guards just completely eliminate a stupid* amount of boilerplate logic in Ruby code

* "stupid" in hindsight, of course

Re: How Discord Scaled Elixir to 5M Concurrent Users

#82

Post by jmcgough »

quaunaut wrote: post #71
jhgg wrote: post #64

Earlier quoted context omitted.

Elixir/Erlang is great, but in addition to knowing the language, there is a huge library of tools within OTP/ERTS that you need to understand to effectively use it and skills learned from other languages don't map too well to Erlang/Elixir as they would to other languages (like going from Python to Go for example). In addition to learning OTP, you also have to learn how a functional language works if you're not famil…

I mean, personally I'm using Elixir more for the language itself than OTP. I have a basic knowledge of OTP now after using the language for 2 years, but FP was my main reason for coming and staying, which for me has been the real fun of it. Then again, FP for me was not difficult at all, and actually made several of the issues I had with my Ruby code not being expressive/obvious enough, or difficulty in composition,…

I fell in love with the language, which is why I kept using it to begin with - working in a functional language is great, and it's very readable and powerful (pattern matching With that said, I could write basically anything Elixir gives me with Ruby. The real value I get from elixir is the blazingly fast performance of BEAM and the speed at which I can build complex and stable distributed systems with OTP.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#83

Post by pmarreck »

Cieplak wrote: post #20

I know that the JVM is a modern marvel of software engineering, so I'm always surprised when my Erlang apps consume less than 10MB of RAM, start up nearly instantaneously, respond to HTTP requests in less than 10ms and run forever, while my Java apps take 2 minutes to start up, have several hundred millisecond HTTP response latency and horde memory. Granted, it's more an issue with Spring than with Java, and Parallel…

> while my Java apps take 2 minutes to start up, have several hundred millisecond HTTP response latency and hoard memory

yeah this is pretty much exactly why I turned away from Clojure and Scala and went for Elixir. Also, super ugly Java stacktraces that were not hard to trigger at all. The vast majority of errors in Elixir are wonderfully explanatory.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#84

Post by pmarreck »

jaequery wrote: post #18

Anyone know if Phoenix/Elixir have something similar to Ruby's bettererror gem? I see Phoenix has a built-in error stack trace page which looks like a clone of bettererror but it doesn't have the real-time console inside of it. Also, I wish they had a ORM like Sequel. These two are really what is holding me back from going full in on Elixir. Anyone can care to comment on this?

Phoenix/Elixir will likely never have an ORM because, well, the O... But I think Ecto is pretty cool, working with it right now on my first Phoenix website, actually

Re: How Discord Scaled Elixir to 5M Concurrent Users

#86

Post by jlouis »

A fun idea is to do away with the "guild" servers in the architecture and simply run message passes from the websocket process over the Manifold system. A little bit of ETS work should make this doable and now an eager sending process is paying for the work itself, slowing it down. This is exactly the behavior you want. If you are bit more sinister you also format most of the message in the sending process and makes it into a binary. This ensures data is passed by reference and not copied in the system. It ought to bring message sends down to about funcall overhead if done right.

It is probably not a solution for current Discord as they rely on linearizability, but I toyed with building an IRCd in Erlang years ago, and there we managed to avoid having a process per channel in the system via the above trick.

As for the "hoops you have to jump through", it is usually true in any language. When a system experiences pressure, how easy it is to deal with that pressure is usually what matters. Other languages are "phase shifts" and while certain things become simpler in that language, other things become much harder to pull off.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#87

Post by jlouis »

alberth wrote: post #85

Is there any update on BEAMJIT? It was super promising 3 or so years ago. But I haven't seen an update. Erlang is amazing in numerous ways but raw performance is not one of them. BEAMJIT is a project to address exactly that. https://www.sics.se/projects/beamjit

Still ongoing work. My personal bet is a bit more on modernizing HiPE however (by using the LLVM backend more).

Re: How Discord Scaled Elixir to 5M Concurrent Users

#88

Post by jlouis »

_asummers wrote: post #74
geogriffin wrote: post #66

Earlier quoted context omitted.

I'm an erlang fan but I wouldn't claim message passing is "fast" in the sense that you might be thinking of (though copying data does have some GC and CPU cache performance benefits that are harder to reason about.) There's no magic in Erlang; copying data is copying data. The benefits of Erlang lie elsewhere, and I've heard Joe Armstrong, when asked about BEAM performance, say something along the lines of "why do yo…

With OTP20, copying data isn't always necessary anymore :)

Copying data is usually necessary in OTP 20 as well I'm afraid. That new optimization doesn't trigger for most of this. But binary data is not copied and hasn't been since at least OTP11 :)

Re: How Discord Scaled Elixir to 5M Concurrent Users

#89

Post by ramchip »

Very interesting article! One thing I'm curious about is how to ensure a given guild's process only runs on one node at a time, and the ring is consistent between nodes.

Do you use an external system like zookeeper? Or do you have very reliable networking and consider netsplits a tolerable risk?

Re: How Discord Scaled Elixir to 5M Concurrent Users

#90

Post by alberth »

jlouis wrote: post #87
alberth wrote: post #85

Is there any update on BEAMJIT? It was super promising 3 or so years ago. But I haven't seen an update. Erlang is amazing in numerous ways but raw performance is not one of them. BEAMJIT is a project to address exactly that. https://www.sics.se/projects/beamjit

Still ongoing work. My personal bet is a bit more on modernizing HiPE however (by using the LLVM backend more).

Amy ETA on when we can start using beamjit?
Post reply on HN