Earlier quoted context omitted.
Off topic, but what's the story behind base-64 encoding your email? Is that just a spam prevention measure?
Yes
I actually really like this idea and am going to steal it. :)
How Discord Scaled Elixir to 5M Concurrent Users
211–220 of 260 posts
Re: How Discord Scaled Elixir to 5M Concurrent Users
#212What is the business model behind Discord? They boast about being free multiple times, how do they make money? Or plan to make money?
They currently have a Nitro service that's $5/month for little features, but I don't know how far that takes them towards profitability.
Re: How Discord Scaled Elixir to 5M Concurrent Users
#213I know that the JVM is a modern marvel of software engineering, so I'm always surprised when my Erlang apps consume less than 10MB of RAM, start up nearly instantaneously, respond to HTTP requests in less than 10ms and run forever, while my Java apps take 2 minutes to start up, have several hundred millisecond HTTP response latency and horde memory. Granted, it's more an issue with Spring than with Java, and Parallel…
> while my Java apps take 2 minutes to start up This is the classic case (and I'm/we are guilty of it as well) of Java apps that are probably traditional monolithic apps that use frameworks with extensive reflection and class loading (ie spring component scanning). You can get ridiculously fast loading time with Java if you use Dart2, a reasonable web framework and no ORMs. I'm talking 500ms... sort of depends on you…
500ms is not "ridiculously fast".
Re: How Discord Scaled Elixir to 5M Concurrent Users
#214Earlier quoted context omitted.
The erlang team has been through 20 major releases, I think I've run all of them since r13 or r14? Being able to upgrade your distributed system is important and they care about making it possible. Generally you upgrade one node at a time, until they're all upgraded and only then can you use new language features. Sometimes you find a build you like and stick with it until something comes up that causes you to upgrad…
> You're going to have the same questions with any other language too. Very few companies get to write clients, servers, and everything else in one language that never updates. Then I don't see much difference between Go, Java, ... vs Erlang except they are simpler to learn plus finding Java, NodeJS, ... devs is much easier. What was the point of using Erlang? It was supposed to solve a problem for us, but we end up…
> Then I don't see much difference between Go, Java, ... vs Erlang except they are simpler to learn plus finding Java, NodeJS, ... devs is much easier.
For the first several years, my team of Erlang devs had zero Erlang experience prior to being hired; they were just smart and flexible developers. I was ahead of the curve because I sort of remembered seeing a Slashdot post about Ericson opening the language way back when.
A lot of people end up using RabbitMQ as their distributed message queue, which is built in Erlang. If you went with that instead of ZeroMQ; and then slowly added more things, there's a reasonable path to writing more Erlang.
I'm not sure if you can really bolt on messaging and reliability features and get the same results; just like bolting on security later, if it's something that you need, it works better if you have it from the beginning.
But certainly, if you're happy with your stack, don't change it.
Re: How Discord Scaled Elixir to 5M Concurrent Users
#215It looks like they have built an interesting, robust and scalable system which is perfectly tailored to their needs. If one didn't want to build all of that in house though, is there anything they've described here that an off the shelf system like https://socketcluster.io doesn't provide ?
Yes. Discord is served over HTTPS. :P (your link is broken; socketcluster.io doesn't serve over HTTPS)
But seriously, Discord actually benchmarked 5 million concurrent users, horizontally distributed, and having to ferry messages across the cluster, with specifically tailored fanout patterns (rather than just a global pub/sub. I.e., who a message goes to varies, rather than just "everyone").
Socketcluster.io only has benchmarks for a single machine, capped at 42k concurrent connections (though to be fair that was due to them running a single client, rather than a limitation of the server). They don't out of the box support horizontal scaling; you're required to spin up your own message queue solution for that.
So, basically, you're advocating a technology that solves -the simplest part of the problem-, and nothing else. Whereas Phoenix + Elixir, even without any of the custom tweaking Discord describes, solve that AND more of the actual problem Discord had. So...yes, and no. Yes, there is plenty here they've describe that is not available in socketcluster.io, but no, nothing they've done here is no generally solved by an off the shelf system, because they're -using- an off the shelf system, Elixir + Phoenix.
Re: How Discord Scaled Elixir to 5M Concurrent Users
#216This writeup make me even more convinced of Elixir becoming one of the large players when it comes to hugely scaling applications. If there is one thing I truly love about Elixir, it is the easiness of getting started, while standing on the shoulders of a giant that is the Erlang VM. You can start by building a simple, not very demanding application with it, yet once you hit a large scale, there is plenty of battle-p…
Truly the strenght of elixir comes for Erlang. I wonder if Scala is also as good as erlang for creating robust and fail proof distributed systems?
The point of Erlang is that it makes writing robust distributed systems easy and fast. The language itself is tiny.
Scala on the other hand is not remotely small and easy to understand.
Re: How Discord Scaled Elixir to 5M Concurrent Users
#217Earlier quoted context omitted.
Erlang (and by extension Elixir) definitely provides a set of tools which are good for building highly concurrent distributed systems. And your systems are likely to have few errors as well as being resilient, if you know what you are doing. But indeed, you need to know your tools like every other part of computer science. Storing 9 billion elements in an array, doing linear search and then complaining your linear se…
Consider for some reason (either technical or political), this company decides to migrate their web socket servers to Akka/Rust/Go/NodeJS. Integrating these new servers into their core cluster is going to be deadly painful. They rely heavily on Erlang/OTP internal clustering. This is not even considered as a challenge in distributed systems but still really painful to implement because of their design decision. This…
That depends what you're going for. Moving web sockets elsewhere are pretty simple regardless of whether you're using Erlang/OTP or not because it's a frontal layer. If you do that all you are doing is creating another layer to hold connections that needs to relay to the rest of the application. The application behind it still has the responsibility to receive and relay everything from those web sockets, determine what message goes to who, etc.
You can move it out just the same as you can elsewhere, you just add an unnecessary layer of complexity and lose a lot of the capabilities already built into the language. It boils down to a simple question of, what would be the purpose behind that decision? What improvement would you get from deciding to move any of that elsewhere? Do you gain speed at the cost of complexity?
Clustering is a non-trivial problem. It can't happen naturally in any language that includes mutable data without explicit oversight and thus most of the needed tooling can't be built into the system and guaranteed to work everywhere. As soon as some part of the system goes from "send message, get response" to "send message, change variable referenced in memory on this machine, get response" you break your ability to naturally cluster.
That leads to a dependency in central relay points for things like setting up web sockets and then having code behind them communicate. The code behind them in whatever language is usually going to be talking to other central relay points like load balancers or pub sub databases rather than directly to other specific servers.
As soon as you have central relay points, they become something else that you have to monitor and scale even if they are very high volume.
There's nothing wrong with this approach and it will certainly work but if you want to avoid that and build a distributed system in another language, you've got to invest a lot in other areas.
Re: How Discord Scaled Elixir to 5M Concurrent Users
#218Earlier quoted context omitted.
Wah. Easy there, Satan :-) That is cool trick though. So it's basically sending the port itself around and changing its ownership, with something like port_connect(Port,NewOwner)? And btw, thank you for writing https://www.erlang-in-anger.com and http://learnyousomeerlang.com !
The trick is more commonly used when writing to sockets. A socket owner is required for reading, not for writing. The trick then is that when you need to write lots of data to a socket to just send a copy of it to the writer so they can dump all their data for cheap, but without changing ownership (which is costly). Also recently I've gotten http://propertesting.com/ out, you might enjoy it :)
Thanks for explaining. I'll have to remember the socket trick.
> Also recently I've gotten http://propertesting.com/ out, you might enjoy it :)
It might be just what I need to understand and start using property tests. I've tried twice and gave up.
Oh and recon! Thanks for that too. Use it almost every day.
Re: How Discord Scaled Elixir to 5M Concurrent Users
#219This writeup make me even more convinced of Elixir becoming one of the large players when it comes to hugely scaling applications. If there is one thing I truly love about Elixir, it is the easiness of getting started, while standing on the shoulders of a giant that is the Erlang VM. You can start by building a simple, not very demanding application with it, yet once you hit a large scale, there is plenty of battle-p…
Truly the strenght of elixir comes for Erlang. I wonder if Scala is also as good as erlang for creating robust and fail proof distributed systems?
Scala paired with Akka is definitely good for that, I don't know how it compares performance wise with Elixir, would be interested to know. We're building a game server backend with lots of messaging in Akka.
Re: How Discord Scaled Elixir to 5M Concurrent Users
#220Earlier quoted context omitted.
Consider for some reason (either technical or political), this company decides to migrate their web socket servers to Akka/Rust/Go/NodeJS. Integrating these new servers into their core cluster is going to be deadly painful. They rely heavily on Erlang/OTP internal clustering. This is not even considered as a challenge in distributed systems but still really painful to implement because of their design decision. This…
That depends what you're going for. Moving web sockets elsewhere are pretty simple regardless of whether you're using Erlang/OTP or not because it's a frontal layer. If you do that all you are doing is creating another layer to hold connections that needs to relay to the rest of the application. The application behind it still has the responsibility to receive and relay everything from those web sockets, determine wh…
> Moving web sockets elsewhere are pretty simple regardless of whether you're using Erlang/OTP or not because it's a frontal layer.
It is supposed to be easy. But they are not using language agnostic messaging. Instead they rely on Erlang distribution protocol. This is what makes it difficult to integrate.
> what would be the purpose behind that decision?
1. To achieve composability. I can replace one web socket server with a C++ implementation to see if I can handle 10 million connections per server. If I fucked it up return back to Erlang implementation otherwise fuck BEAM. I'm moving to native code. The beauty of it is that I can do it one step at a time. Even I can implement web socket layer in multiple languages and experiment all possible options in production and no one even notices it.
2. I don't have to pay for 20 Erlang guys who are mostly senior devs. Get 5 of them to write the core and the rest of the team NodeJS guys.