Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Discord Scaled Elixir to 5M Concurrent Users

blog.discordapp.com

Post reply on HN
Print view View original

91–100 of 260 posts

Re: How Discord Scaled Elixir to 5M Concurrent Users

#91

Post by _asummers »

jlouis wrote: post #88
_asummers wrote: post #74

Earlier quoted context omitted.

With OTP20, copying data isn't always necessary anymore :)

Copying data is usually necessary in OTP 20 as well I'm afraid. That new optimization doesn't trigger for most of this. But binary data is not copied and hasn't been since at least OTP11 :)

The release says "Erlang literals". Wouldn't that be things like atoms, integers, booleans, tuples, and the like? That plus binary data should cover a good deal, unless I'm reading too much into the blurb on the release notes.

Under what circumstances does it get triggered (since you seem to be more knowledgable about this than I am!)? I expect records would not fall under this.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#92

Post by jlouis »

rdtsc wrote: post #5

Good stuff. Erlang VM FTW! > mochiglobal, a module that exploits a feature of the VM: if Erlang sees a function that always returns the same constant data, it puts that data into a read-only shared heap that processes can access without copying the data There is a nice new OTP 20.0 optimization - now the value doesn't get copied even on message sends on the local node. Jesper L. Andersen (jlouis) talked about it in h…

It isn't that likely the OTP20 optimization helps here. If the process never sends a message containing the literal value, then there is no benefit in the optimization. What `mochiglobal` and friends are good at is when you have a large set of data (A ring, say) which update rarely, so you can treat it as semi-static data in the system. But then you shouldn't really send that ring data around in the system too much, although it will now be free. [There is a nice subscription-based approach to updates which are now feasible in OTP20, but that is more for convenience]

if send/2 takes 30us to 70us, I'm guessing blocking as well, either on distributed communication or something else along those lines. For local message passes to take that long, my something-is-amiss-sixth-sense is tingling.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#93

Post by mononcqc »

jlouis wrote: post #86

A fun idea is to do away with the "guild" servers in the architecture and simply run message passes from the websocket process over the Manifold system. A little bit of ETS work should make this doable and now an eager sending process is paying for the work itself, slowing it down. This is exactly the behavior you want. If you are bit more sinister you also format most of the message in the sending process and makes…

The true evil approach is to send the socket around, not the message, so that there is no copying required no matter what ;)

Re: How Discord Scaled Elixir to 5M Concurrent Users

#94

Post by bigtunacan »

vertex-four wrote: post #57
bigtunacan wrote: post #27

Earlier quoted context omitted.

When Rails arrived on the scene it was very different from everything else, but also there weren't 1000 new languages/ frameworks popping up all at once. I'm not implying in anyway that Elixir is bad. I just think there are too many horses these days to know which to bet on. Elixir? Node? Go? Rust? Something else?

Depends on what you're doing, really. Rust is lovely for security-sensitive code - I'm writing a customer identity management system with Rust+Postgres+Redis. Go is... acceptable... for "glue" code where PHP would've been used a decade ago and Perl before that - all the successes of it I've seen fall into that sort of pattern. Elixir is great when you need to think about networked, stateful systems on the scale of a…

I understand where they are supposed to fit, but the problem is there are so many existing tools to fill the same needs already.

As an example Erlang and Elixir both fit essentially the same bill so it seems to me what was already a small niche is just getting fractured.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#95

Post by ramchip »

alberth wrote: post #85

Is there any update on BEAMJIT? It was super promising 3 or so years ago. But I haven't seen an update. Erlang is amazing in numerous ways but raw performance is not one of them. BEAMJIT is a project to address exactly that. https://www.sics.se/projects/beamjit

Yes, there's updates:

http://www.erlang-factory.com/sfbay2017/lukas-larson.html

Re: How Discord Scaled Elixir to 5M Concurrent Users

#96

Post by jlouis »

alberth wrote: post #90
jlouis wrote: post #87

Earlier quoted context omitted.

Still ongoing work. My personal bet is a bit more on modernizing HiPE however (by using the LLVM backend more).

Amy ETA on when we can start using beamjit?

Given that it has been postponed a couple of times, no. JITs are hard to pull off and it will probably have a period of worse stability as well before it matures. Another problem is getting a JIT to be faster than the interpreter. Erlang's BEAM is threaded code and also macro-instructions, so it almost looks like a JIT internally.

The big gains would be in inlining across module boundaries and type speculation. But I hold that if we could compile bundles of modules in HiPE, we would have the same gain for a fraction of the development and maintenance effort.

The biggest lure of native code generation would be that we could get rid of a lot of C code in the system as the native cogen would be able to rival the C code in speed. Many Erlang programs spend shockingly little time in the emulator loop, especially if they are communication heavy.

If you need speed today, don't underestimate a port-program. My test is that you can pipeline about a million requests back and forth to an OCaml program per second per core. So if your work is on the order of 1+ milliseconds, this is usually a feasible strategy. Espcially because OS isolation means you can handle exceptions in the OCaml program from the Erlang side by restarting the port.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#97

Post by _asummers »

jmcgough wrote: post #82
quaunaut wrote: post #71

Earlier quoted context omitted.

I mean, personally I'm using Elixir more for the language itself than OTP. I have a basic knowledge of OTP now after using the language for 2 years, but FP was my main reason for coming and staying, which for me has been the real fun of it. Then again, FP for me was not difficult at all, and actually made several of the issues I had with my Ruby code not being expressive/obvious enough, or difficulty in composition,…

I fell in love with the language, which is why I kept using it to begin with - working in a functional language is great, and it's very readable and powerful (pattern matching With that said, I could write basically anything Elixir gives me with Ruby. The real value I get from elixir is the blazingly fast performance of BEAM and the speed at which I can build complex and stable distributed systems with OTP.

The value to me (other than OTP + friends) is not what Elixir gives you, but what it takes away. Not being able to monkey patch things and change mutable state is really freeing. You can do it if you're diligent, of course (I code Java with as many static methods and final, immutable fields as I can and what I know about Ruby, it has things like freeze), but it's hard to never reach for a tool "just this once" if you're in a time crunch.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#98

Post by jlouis »

_asummers wrote: post #91
jlouis wrote: post #88

Earlier quoted context omitted.

Copying data is usually necessary in OTP 20 as well I'm afraid. That new optimization doesn't trigger for most of this. But binary data is not copied and hasn't been since at least OTP11 :)

The release says "Erlang literals". Wouldn't that be things like atoms, integers, booleans, tuples, and the like? That plus binary data should cover a good deal, unless I'm reading too much into the blurb on the release notes. Under what circumstances does it get triggered (since you seem to be more knowledgable about this than I am!)? I expect records would not fall under this.

A "literal" in this case is a constant value defined in a module. Those live in a separate space in the VM and are referenced directly because they are immutable and can be shared. If you sent such a literal before OTP20, it would be copied into the heap of the target process. Not anymore.

But it doesn't help with cases where you are constructing a term (dynamically) in a process and sending that term. There is more meat in the blog post of mine: https://medium.com/@jlouis666/an-erlang-otp-20-0-optimizatio...

Re: How Discord Scaled Elixir to 5M Concurrent Users

#99

Post by vertex-four »

bigtunacan wrote: post #94
vertex-four wrote: post #57

Earlier quoted context omitted.

Depends on what you're doing, really. Rust is lovely for security-sensitive code - I'm writing a customer identity management system with Rust+Postgres+Redis. Go is... acceptable... for "glue" code where PHP would've been used a decade ago and Perl before that - all the successes of it I've seen fall into that sort of pattern. Elixir is great when you need to think about networked, stateful systems on the scale of a…

I understand where they are supposed to fit, but the problem is there are so many existing tools to fill the same needs already. As an example Erlang and Elixir both fit essentially the same bill so it seems to me what was already a small niche is just getting fractured.

Elixir is just alternative (Ruby-ish) syntax for Erlang, as much as people have hyped it up - Erlang code can call Elixir code and vice versa with essentially no abstraction cost. In fact, the most popular web framework for Elixir is heavily built on Erlang code.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#100

Post by whitepoplar »

josevalim wrote: post #67
janekm wrote: post #46

Earlier quoted context omitted.

I read it a little differently. The whole article is about how Erlang/Elixir fails at its core reason for existence (fast message passing between distributed processes) and all the complicated work-arounds they had to implement to avoid actually using this core feature of Erlang.

People tend to forget that scalability is not a binary property. You always scale up to some users, up to some architecture, up to some amount of nodes. There is no system that will scale to infinity without requiring developer intervention once business needs and application patterns start to settle in. Distributed Erlang/Elixir has known limitations . For example, the network is fully meshed, which gives you about…

This is a great point! What would you say is the best book to really learn OTP?
Post reply on HN