Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Discord Scaled Elixir to 5M Concurrent Users

blog.discordapp.com

Post reply on HN
Print view View original

161–170 of 260 posts

Re: How Discord Scaled Elixir to 5M Concurrent Users

#161

Post by josevalim »

majidazimi wrote: post #156
josevalim wrote: post #67

Earlier quoted context omitted.

People tend to forget that scalability is not a binary property. You always scale up to some users, up to some architecture, up to some amount of nodes. There is no system that will scale to infinity without requiring developer intervention once business needs and application patterns start to settle in. Distributed Erlang/Elixir has known limitations . For example, the network is fully meshed, which gives you about…

What difference does it make? Erlang/OTP distribution doesn't have pluggable architecture. Sooner or later you will reach a point that you have to modify it. Then you are diverging from the original branch which makes it even more difficult to maintain it. You have to merge your additions into every release (minor or major) and test it thoroughly. A better architecture for a distributed system has a strong composabil…

What do you mean by architecture here? If you mean the roles different nodes take and their topology, I actually doubt you can be decoupled from your architecture in a distributed system because they directly affect your design and capabilities.

You can move away from fully meshed for topologies but how does this choice affect node ups and node downs? Rebalancing can affect how you store data in the cluster.

How many connections should you have between nodes? A single connection makes the ordering guarantee straight-forward. Multiple connections is more performance but requires care if you need ordering and is more efficiently done along side your application.

And what about process placement? On which side of CAP do you want your registries to sit?

If you and your team is capable of "writing your own standard well documented distributed layer" upfront, then you are in a better position than most to take those decisions. But writing a distributed system is hard, so I will gladly start with a well-designed system, especially at the beginning of the project, when it may be unclear which patterns I will need as my application and business grow.

And most times, it will be good enough.

As far as OTP goes, you can plug your own discovery/topology mechanism as well as your own module for handling the connection between nodes. But, as mentioned in my previous reply, some of those issues may be better solved on the side, e.g. a different tcp/udp connection for data transmitting.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#162

Post by jhgg »

ramchip wrote: post #89

Very interesting article! One thing I'm curious about is how to ensure a given guild's process only runs on one node at a time, and the ring is consistent between nodes. Do you use an external system like zookeeper? Or do you have very reliable networking and consider netsplits a tolerable risk?

We use etcd.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#163

Post by OOPMan »

5 million concurrent users is great and all, but it would be nice if Discord could work out how to use WebSockets without duplicating sent messages.

This seems to happen a lot when you are switching between wireless networks (E.g. My home router has 2Ghz and 5Ghz wireless networks) or when you're on mobile (Seems to happen regularly, even if you're not moving around).

It's terribly annoying though and makes using the app via the mobile client to be very tedious.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#164

Post by k__ »

OOPMan wrote: post #163

5 million concurrent users is great and all, but it would be nice if Discord could work out how to use WebSockets without duplicating sent messages. This seems to happen a lot when you are switching between wireless networks (E.g. My home router has 2Ghz and 5Ghz wireless networks) or when you're on mobile (Seems to happen regularly, even if you're not moving around). It's terribly annoying though and makes using the…

Coming from IRC with netsplits and stuff, this seems like a first world problem to me hehe

Re: How Discord Scaled Elixir to 5M Concurrent Users

#165

Post by toast0 »

majidazimi wrote: post #154

It seems awkward to me. What if Erlang/OTP team can not guarantee message serialization compatibility across a major release? How you are going to upgrade a cluster one node at a time? What if you want to communicate with other platforms? How you are going to modify distribution protocol on a running cluster without downtime? As soon as you introduce standard message format, then all nice features such as built-in di…

The erlang team has been through 20 major releases, I think I've run all of them since r13 or r14? Being able to upgrade your distributed system is important and they care about making it possible. Generally you upgrade one node at a time, until they're all upgraded and only then can you use new language features. Sometimes you find a build you like and stick with it until something comes up that causes you to upgrade.

There are many ways to interoperate with other languages. Including libraries for other languages to claim to be erlang nodes (they will have to upgrade too when you want to upgrade to a newer version of erlang with distribution protocol changes, for example when maps we're introduced in r17).

You can easily do standard dist messaging within your erlang cluster and serialize to whatever makes sense at the boarder. You can serialize to erlang term format, if you like; it's well specified, but not terribly compact.

You're going to have the same questions with any other language too. Very few companies get to write clients, servers, and everything else in one language that never updates.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#166

Post by josevalim »

majidazimi wrote: post #159
andy_ppp wrote: post #157

Earlier quoted context omitted.

I think, 1) you are misunderstanding the grand parents points. Better to have primitives that scale you to X rather than having to get to X on your own. 2) The grand parent wrote the Elixir language. 3) Making sweeping statements like "A better architecture for a distributed system has a strong composability property" is very easy.

> 1) Better to have primitives that scale you to X rather than having to get to X on your own. For a fast time to market, I do agree with you. But if you are well established company in the market, then in most cases "do it yourself" is the best idea. Unless you have the money to call for experts to fix the problem for you. > 2) The grand parent wrote the Elixir language. I didn't know him, nor I care. > 3) Making sw…

> But if you are well established company in the market, then in most cases "do it yourself" is the best idea. Unless you have the money to call for experts to fix the problem for you.

This requires having distributed system experts on your team or having the money to call them from day one, before you even reach the market and before you are even sure you will have custom needs. If you have the need, the expertize and the time, then surely. But writing a distributed system from scratch should certainly not be the first choice (IMO).

Re: How Discord Scaled Elixir to 5M Concurrent Users

#167

Post by sametmax »

janekm wrote: post #46
iagooar wrote: post #19

This writeup make me even more convinced of Elixir becoming one of the large players when it comes to hugely scaling applications. If there is one thing I truly love about Elixir, it is the easiness of getting started, while standing on the shoulders of a giant that is the Erlang VM. You can start by building a simple, not very demanding application with it, yet once you hit a large scale, there is plenty of battle-p…

I read it a little differently. The whole article is about how Erlang/Elixir fails at its core reason for existence (fast message passing between distributed processes) and all the complicated work-arounds they had to implement to avoid actually using this core feature of Erlang.

Me too. I mean, 5 millions is not that many users. It requires work, but it's not Google size by any mean. It's just a successful service.

Give that the whole selling point of Erlang/Elixir is scalability at the price of the rest, the article is really telling me to avoid the tech.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#168

Post by majidazimi »

ramchip wrote: post #160
majidazimi wrote: post #154

It seems awkward to me. What if Erlang/OTP team can not guarantee message serialization compatibility across a major release? How you are going to upgrade a cluster one node at a time? What if you want to communicate with other platforms? How you are going to modify distribution protocol on a running cluster without downtime? As soon as you introduce standard message format, then all nice features such as built-in di…

Distributed Erlang compatibility is guaranteed for not just one, but two major versions. You can do a standard rolling upgrade and the newer nodes will talk to older nodes just fine. http://erlang.org/pipermail/erlang-questions/2011-June/05946... Foreign nodes in C, Java, Python, etc. can also join an Erlang cluster: http://erlang.org/doc/tutorial/cnode.html http://erlang.org/doc/apps/jinterface/jinterface_users_guid…

It's great that they support C and JVM stack. But for other major platforms such as NodeJS, Go, ... we still need to approach the goal with some C extensions integrated in both platforms. Erlang/OTP distribution/clustering is really not designed for heterogeneous environment which is fine, since it is used in telecom industry with nice commercial support contracts backed by Ericsson. However I don't consider it as an amazing piece of engineering from a perspective of a system designer who is dealing with many teams with different language/stack preferences.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#169

Post by passer-by-123 »

sametmax wrote: post #167
janekm wrote: post #46

Earlier quoted context omitted.

I read it a little differently. The whole article is about how Erlang/Elixir fails at its core reason for existence (fast message passing between distributed processes) and all the complicated work-arounds they had to implement to avoid actually using this core feature of Erlang.

Me too. I mean, 5 millions is not that many users. It requires work, but it's not Google size by any mean. It's just a successful service. Give that the whole selling point of Erlang/Elixir is scalability at the price of the rest, the article is really telling me to avoid the tech.

5 million concurrent users, connected at the same time, sending and receiving data through a persistent connection. Phoenix itself got 2 million on a single node: http://www.phoenixframework.org/blog/the-road-to-2-million-w...

They likely have much more than 5 million users.

To put this amount in perspective, you get 5 million connections after receiving 3000 connections per second, which are never dropped and remain connected, for ~28 minutes. The majority of websites do not get even 300 requests per second.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#170

Post by di4na »

bsder wrote: post #38
iagooar wrote: post #19

This writeup make me even more convinced of Elixir becoming one of the large players when it comes to hugely scaling applications. If there is one thing I truly love about Elixir, it is the easiness of getting started, while standing on the shoulders of a giant that is the Erlang VM. You can start by building a simple, not very demanding application with it, yet once you hit a large scale, there is plenty of battle-p…

> I see a bright future for Elixir, and a breath of fresh air for Erlang. Is Erlang really that much of a barrier? Erlang was a touch odd, but I didn't find the language itself that mind-bending. Wrapping my head around the proper way to structure things and the proper use of the OTP libraries was much more time-consuming. That level of architectural thought doesn't magically go away because you changed the language.…

The problem is not and has never been the syntax. It is the tooling. No build tool. No way to generate a new project. Do it yourself docs. No templating system. Macros ala C. No package manager. Building releases was a dark art.

It is now getting better. But it was a really steep curve to adoption.

Post reply on HN