Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Discord Scaled Elixir to 5M Concurrent Users

blog.discordapp.com

Post reply on HN
Print view View original

111–120 of 260 posts

Re: How Discord Scaled Elixir to 5M Concurrent Users

#111

Post by tmat »

mahyarm wrote: post #36
jakebasile wrote: post #16

I'm continually impressed with Discord and their technical blogs contribute to my respect for them. I use it in both my personal life (I run a small server for online friends, plus large game centric servers) and my professional life (instead of Slack). It's a delight to use, the voice chat is extremely high quality, text chat is fast and searchable, and notifications actually work. Discord has become the de facto pl…

They could side step to corporate slack on-prem chat style solutions and be a pretty strong contender. They were scaling huge chat rooms and such way better than slack was a year ago. But they seem to refuse suggestions to do so continuously, so they seem to have some business plan somewhere.

I discussed moving my companies chat to discord because its a better experience than Slack in every way. They wouldn't go for it because it's marketing is so gamer oriented and wouldn't look professional when we invite clients onboard :(

Re: How Discord Scaled Elixir to 5M Concurrent Users

#112

Post by tmat »

didibus wrote: post #60

So, at this point, every language was scaled to very high concurrent loads. What does that tell us? Sounds to me like languages don't matter for scale. In fact, that makes sense, scale is all about parallel processes, horizontally distributing work can be achieved in all language. Scale is not like perforance, where if you need it, you are restricted to a few languages only. That's why I'd like to hear more about pro…

Yes, but I can run a Elixir app at scale far far far cheaper than I can a Ruby app.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#113

Post by jjirsa »

Vishnevskiy wrote: post #50
joonoro wrote: post #41

Earlier quoted context omitted.

Thanks for the response, it's good to know what's causing the problems with messages and that it's being worked on. I'll try to contact support next time I have voice issues with my group.

:) np We love Cassandra, and hate it at the same time. Check out this nasty bug we got. https://issues.apache.org/jira/browse/CASSANDRA-13004

Two things are true about Cassandra:

It is by far the best at doing what it does

It has plenty of room for improvement

Re: How Discord Scaled Elixir to 5M Concurrent Users

#114

Post by _asummers »

ramchip wrote: post #109
rubyn00bie wrote: post #58

Earlier quoted context omitted.

Just look for Erlang tools, there are loads available. The only problem with Elixir right now is sometimes people forget to look at Erlang for already solved problems/tooling/etc. Erlang/BEAM has been alive and well in some very serious mission critical applications for 20 years. Just look at how telecoms used it (its origin).

I don't think that's the problem for metrics. Exometer is the go-to Erlang library, but its dependencies are a nightmare (e.g. https://github.com/Feuerlabs/exometer/issues/154 or https://github.com/Feuerlabs/exometer#dependency-management ), it's essentially unmaintained, and integrating it with Ecto or Phoenix is roll-your-own. I think there is very much a need for a more modern library that plays nicely with Hex.

We've had success with Elixometer, though the dependency pinning was a pain. All but two are fully on Hex now.

https://gist.github.com/asummers/3479eb6e3914d9b9a24c9566468...

Re: How Discord Scaled Elixir to 5M Concurrent Users

#116

Post by bigtunacan »

vertex-four wrote: post #99
bigtunacan wrote: post #94

Earlier quoted context omitted.

I understand where they are supposed to fit, but the problem is there are so many existing tools to fill the same needs already. As an example Erlang and Elixir both fit essentially the same bill so it seems to me what was already a small niche is just getting fractured.

Elixir is just alternative (Ruby-ish) syntax for Erlang, as much as people have hyped it up - Erlang code can call Elixir code and vice versa with essentially no abstraction cost. In fact, the most popular web framework for Elixir is heavily built on Erlang code.

That was my point exactly. Before if you really needed the benefits of running on BEAM then you would have chosen Erlang. Now that subsection will fragment between Erlang and Elixir. Today Elixir developers still will lean heavily on the interaction of Erlang libraries until someone in Elixir-land needs something that isn't supported so then they will re-invent that particular wheel in Erlang and thus the cycle continues ever on.

Then as you stated yourself Go is being used in areas where PHP & Perl were used before (as were also Ruby, Python, and Lua). So now there is just one more fracture there. Of course this is especially interesting since Go was supposed to be a better C or C++ then that should have placed it pretty squarely in systems programming land, yet is has managed to gain more traction as a glue and web services languages, ergo it ends up competing in this space when it probably should have competed more with C/C++/Rust.

Rust; as a systems programming language still has to compete with C/C++/ObjC in this space and considering that "all the systems" already run on these languages that is one huge challenge to change the guard there. I'm not saying that Rust is bad, rather just that I think it's long term outlook may actually be rather bleaker than other languages due to these challenges. Quite frankly D had some great concepts and improvements as a systems language over C++, but it is still barely a blip on anyone's radar.

If an individual is just trying to learn and expand their horizons then any of these languages are great to pick up. If, on the other hand, they are banking their future career on one then none is a sure fire bet right now. One would honestly still have more luck with a tried and true like Java.

Personally I enjoy picking up new languages just to see things from a different perspective and continue learning. Heck I spend much of my free time coding in Crystal which hasn't even hit a 1.0 launch so that adds pretty much zero improvement to my career prospects. :)

Re: How Discord Scaled Elixir to 5M Concurrent Users

#117

Post by rozap »

jlouis wrote: post #96
alberth wrote: post #90

Earlier quoted context omitted.

Amy ETA on when we can start using beamjit?

Given that it has been postponed a couple of times, no. JITs are hard to pull off and it will probably have a period of worse stability as well before it matures. Another problem is getting a JIT to be faster than the interpreter. Erlang's BEAM is threaded code and also macro-instructions, so it almost looks like a JIT internally. The big gains would be in inlining across module boundaries and type speculation. But I…

Would you say the OCaml-port-as-an-optimization-strategy only makes sense if the program is compute bound, though?

The reason I ask is because we're running a Erlang+JInterface program and the performance advantage the JVM has over BEAM is less than I would have expected. Even batching requests up into big pieces, we still see it's about 30% slower than running the same stuff in Elixir, without so much copying. But the reason we're doing JVM stuff at all is so we can re-use a whole bunch of code we already had written, and I would have expected it would have been a marginal performance win as well, but it's not.

Perhaps we're doing it wrong, too.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#118

Post by cdegroot »

toast0 wrote: post #108
janekm wrote: post #46

Earlier quoted context omitted.

I read it a little differently. The whole article is about how Erlang/Elixir fails at its core reason for existence (fast message passing between distributed processes) and all the complicated work-arounds they had to implement to avoid actually using this core feature of Erlang.

Erlang's core reason for existence is to control telephone switches, which had two independent general purpose computers connected to the physical switch. So reliability, redundancy, recovery, and fault isolation were the core needs; that drove the design for isolated processes with message passing between them. Because Erlang was in the control plane, and only managing the signal path, not passing the signals itself…

That, indeed. When I compare Elixir/Erlang to some other systems I worked on, "shallow" is the word that pops up. You hit a limitation, you dig into some source code, and you find out that it's pretty simple to understand and to fix it. It feels manageable, I've yet have to run into frustrating roadblocks, and that all gives me the confidence that when I do need to scale up, I have a system I will understand and will be able to adapt. It looks like Discord's story confirms that.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#119

Post by Vishnevskiy »

jjirsa wrote: post #113
Vishnevskiy wrote: post #50

Earlier quoted context omitted.

:) np We love Cassandra, and hate it at the same time. Check out this nasty bug we got. https://issues.apache.org/jira/browse/CASSANDRA-13004

Two things are true about Cassandra: It is by far the best at doing what it does It has plenty of room for improvement

You are 100% right that it is the best at what it does.

And thank you again with helping us with that bug :) we really appreciate it.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#120

Post by Vishnevskiy »

lpgauth wrote: post #105
Vishnevskiy wrote: post #31

Earlier quoted context omitted.

The messages struggles have been sadly due to issues with Cassandra and GC pauses caused by bugs within it. We have been trying to work with the Cassandra developers to resolve these. Voice issues should not be happening. Please contact our support with more information and we will gladly investigate.

You might want to have a look at ScyllaDB.

We are currently in the process of testing ScyllaDB with double writes for our fixed data clusters. It is very scary to transition to something so new :)

Our message storage clusters have a very large set of data that keeps increasing and using Scylla without incremental repair will suck so we are waiting on that.

Post reply on HN