Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Discord Scaled Elixir to 5M Concurrent Users

blog.discordapp.com

Post reply on HN
Print view View original

251–260 of 260 posts

Re: How Discord Scaled Elixir to 5M Concurrent Users

#251

Post by gregpardo »

manukall wrote: post #158
toolz wrote: post #37

Earlier quoted context omitted.

I think the next big push in elixir is tooling around metrics. Once there are hard numbers to market to non-techies I think we'll see a large shift towards elixir.

I've recently launched https://pryin.io , an application performance monitoring tool made for Elixir and Phoenix. It hooks into Phoenix and gives you insights into how long your request / channels take and what Ecto queries are run / how long those take. You can also manually augment pretty much anything else (background jobs, API calls, ...). Plus it keeps track of some important BEAM metrics like memory consumption…

Looks cool. Might want to have a demo site running for people to poke around.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#252

Post by lostcolony »

jondubois wrote: post #248
lostcolony wrote: post #247

Earlier quoted context omitted.

That's perfectly fair; fix the marketing then. :P In evaluating a solution, the marketing is the -first thing- anyone looks at. And how it currently reads, "SocketCluster only provides you the interface for synchronizing instance channels - It doesn't care what technology/implementation you use behind the scenes to make this work" definitely reads as "You need a technology/implementation behind the scene" rather than…

Thanks for the advice. Yes, it handles message passing between instances in the cluster. That means if you publish a message on a channel whilst connected to one host, the message will also reach subscribers to that channel which are on any other host in the cluster. It shards all channels across available brokers, when you scale up the number of brokers, it will automatically migrate the shards across available brok…

Okay, nice. Then yeah, given some performance benchmarks showing some numbers at increasing number of nodes, with messages being broadcast across 1-to-1 channel pairings (i.e., direct message), 1-to-100 (for groups), and 1-to-all, I think it could sell itself as a pretty compelling turnkey solution (barring the scc-state concern which you're aware of).

Possibly also consider some testing and documentation around geographic distribution; what happens if the nodes are located in different datacenters with non-trivial latency between them? Is that an issue? In the event of netsplits, does it split brain (probably not, given scc-state, but addressing that might cause it to)? That might be fine, it might not, depending on the use case, and just documenting what happens (by default, at least, if it's to be tunable) would be helpful as well.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#253

Post by lomnakkus »

rdtsc wrote: post #249
lomnakkus wrote: post #243

Earlier quoted context omitted.

> It's not a superset until it has a non-sharable memory heaps between threads, complete and easy hot code reloading, dynamic tracing (being able to log into a node and update code at will as it the application is running). The first two I can definitely see -- particularly for robustness and debugging -- but I'm a bit surprised by the last one. Do people actually really log into running production systems and update…

> Do people actually really log into running production systems and update code Yes, I've done it once in a while. Cases could be is to deploy a fix and the customer's system is up and running. If say it's something urgent that can't wait until it goes through the full deployment pipeline. Because hot code reloading works so well in Erlang it's not risky as doing it in Java for example. In fact upgrading by hot code…

Ah, right, I guess the "add logging" case is a pretty compelling and not-too-scary (for my sensibilities) one. Point also well taken about it being safer because of the shared-nothing nature of processes.

Thanks for the perspective!

Re: How Discord Scaled Elixir to 5M Concurrent Users

#254

Post by defined »

rdtsc wrote: post #249
lomnakkus wrote: post #243

Earlier quoted context omitted.

> It's not a superset until it has a non-sharable memory heaps between threads, complete and easy hot code reloading, dynamic tracing (being able to log into a node and update code at will as it the application is running). The first two I can definitely see -- particularly for robustness and debugging -- but I'm a bit surprised by the last one. Do people actually really log into running production systems and update…

> Do people actually really log into running production systems and update code Yes, I've done it once in a while. Cases could be is to deploy a fix and the customer's system is up and running. If say it's something urgent that can't wait until it goes through the full deployment pipeline. Because hot code reloading works so well in Erlang it's not risky as doing it in Java for example. In fact upgrading by hot code…

You mentioned tracing. I'd like to expand on that a little.

What many people may be interested to know about Erlang is that you can log in to a production system, start a new shell running a tracer that listens on a localhost TCP socket[1], and use the dbg module in the production VM to trace calls and messages (and more) between any functions in any processes - in the running production system - and send them to the tracer node.

Done judiciously, the overhead is negligable, and the benefits are great. You can zoom in on bugs in real time.

I find the syntax of dbg match specs to be ugly, but it has saved my bacon so often it is so worth it, and it doesn't get mentioned that much, even though I feel it is almost as much a superpower as hot code loading.

[1] You use the separate shell to avoid accidentally crashing the production VM; if you do something boneheaded in the port-based shell, you can kill it and the production VM will just stop sending trace data to the dead TCP socket.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#255

Post by eggy »

pmarreck wrote: post #191
eggy wrote: post #140

Earlier quoted context omitted.

I tried translating some of the book "The Handbook of Neuroevolution Through Erlang" into LFE (Lisp Flavored Erlang), and Elxir (somebody's done this already I believe). Robert Virding, a co-creator of Erlang, creator of LFE, also wrote Luerl - Lua implemented in Erlang. So if syntax is important you have many choices to use the BEAM VM/OTP in a Ruby-like language (Elixr), a Lisp (LFE) or in Lua (Luerl)! I am now loo…

How far did you get with the "Neuroevolution Through Erlang" book? I happen to own it (there must be like 10 of us!) but haven't gotten around to it yet...

I am more than a third of the way through. I got distracted, and need to pick it up again. I like the writing style, and that if focuses on TWEANNs and practical examples rather than broad ML. I like the BEAM/OTP but Erlang syntax rubs me the wrong way. I've tried Elixir and LFE, and now I might try Luerl. Not enough time in the day...

Re: How Discord Scaled Elixir to 5M Concurrent Users

#256

Post by yellowapple »

amorphid wrote: post #138
pmarreck wrote: post #83

Earlier quoted context omitted.

> while my Java apps take 2 minutes to start up, have several hundred millisecond HTTP response latency and hoard memory yeah this is pretty much exactly why I turned away from Clojure and Scala and went for Elixir. Also, super ugly Java stacktraces that were not hard to trigger at all. The vast majority of errors in Elixir are wonderfully explanatory.

A lot of the Erlang-y error messages that bubble up into Elixir can get pretty gnarly. Earlier today I accidentally tried doing `Map.get(key, map)` (map should be first arg) in a `&handle_call/1` callback on a GenServer module, and it got ugly fast. I knew what the problem was from experience, but the Erlang-y error can be intimidating to see for the first time.

Sometimes you have to jump into the source of some dependency (or even OTP itself) to really understand what's causing the error.

Thankfully, Erlang and Elixir tend to make this a lot less painful, in no small part due to their respective declarative/functional programming traits.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#257

Post by oldpond »

whitepoplar wrote: post #100
josevalim wrote: post #67

Earlier quoted context omitted.

People tend to forget that scalability is not a binary property. You always scale up to some users, up to some architecture, up to some amount of nodes. There is no system that will scale to infinity without requiring developer intervention once business needs and application patterns start to settle in. Distributed Erlang/Elixir has known limitations . For example, the network is fully meshed, which gives you about…

This is a great point! What would you say is the best book to really learn OTP?

Pragprog has a good selection. https://pragprog.com/titles/category/elixir?f[include_mine]=...

Re: How Discord Scaled Elixir to 5M Concurrent Users

#258

Post by oldpond »

When have you ever read, "How Acme scaled J2EE to 5M Concurrent Users"? I became an IT architect in 1998 at IBM, the year Sun released j2ee and IBM released Websphere. I have experienced 20 years of enterprise Java and object oriented computing, and I was thrilled when Elixir came out. I was a mainframe programmer before OO became all the rage, so I never really felt at home doing objects. Functional programming feels completely natural to me though.

What I like about this article is that they shared everything they learned with the community. Thank you for that excellent experience report.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#259

Post by OvermindDL1 »

tormeh wrote: post #237
justinhj wrote: post #219

Earlier quoted context omitted.

Scala paired with Akka is definitely good for that, I don't know how it compares performance wise with Elixir, would be interested to know. We're building a game server backend with lots of messaging in Akka.

Scala/Akka is a lot faster than Erlang, but it also lacks per-actor GC, which means that latency is going to be higher in a Scala/Akka system. Also, the per-actor heaps in Beam means there's no risk of two Erlang actors having a reference to the same memory, whereas that's easy to do in Scala/Akka.

Actually you'd be surprised at just how fast the BEAM passes messages, especially as it also includes linking, monitoring, cross-process, cross-system, among other features. When creating a multi-thread atomic messaging system in C++ long ago I was able to out-perform the BEAM (though I'd not rank it as significantly out-perform), but once I added failure handling features among others then it never got close to BEAM's speed again in raw message passing.

However, the BEAM is not very fast on executing actual code, it is like Python in that way where it is good to slave out CPU-heavy work; the BEAM is built for async IO, and yes, running a JVM (or C++) system as a node on a BEAM mesh is fantastic for that (or a port, or NIF for small work).

Re: How Discord Scaled Elixir to 5M Concurrent Users

#260

Post by OvermindDL1 »

eggy wrote: post #255
pmarreck wrote: post #191

Earlier quoted context omitted.

How far did you get with the "Neuroevolution Through Erlang" book? I happen to own it (there must be like 10 of us!) but haven't gotten around to it yet...

I am more than a third of the way through. I got distracted, and need to pick it up again. I like the writing style, and that if focuses on TWEANNs and practical examples rather than broad ML. I like the BEAM/OTP but Erlang syntax rubs me the wrong way. I've tried Elixir and LFE, and now I might try Luerl. Not enough time in the day...

For note, luerl is more of a library for erlang/elixir/BEAM and not a full language, as it is basically just lua's interpreter rewritten in erlang with some modifications to make it fit the system a bit better than normal Lua (there is normal Lua available via a Port though).
Post reply on HN