Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Discord Scaled Elixir to 5M Concurrent Users

blog.discordapp.com

Post reply on HN
Print view View original

201–210 of 260 posts

Re: How Discord Scaled Elixir to 5M Concurrent Users

#201

Post by jlouis »

majidazimi wrote: post #154

It seems awkward to me. What if Erlang/OTP team can not guarantee message serialization compatibility across a major release? How you are going to upgrade a cluster one node at a time? What if you want to communicate with other platforms? How you are going to modify distribution protocol on a running cluster without downtime? As soon as you introduce standard message format, then all nice features such as built-in di…

You are missing that major-release compatibility is something which is taken seriously. Usually you have opcode stability for at least one release so OTP20 can run OTP19 bytecode. But not vice versa, naturally.

The same is true for serialization: new features are often introduced and then put to use a couple of major releases later. This ensures backwards compatibility. If you couldn't upgrade your cluster one node at a time, safely, then you would have to stop the system. The serialization format is also built to be machine-agnostic: You can run data from a 32bit little endian windows machine to a 64bit big endian sparc machine if you want, and it will work seamlessly. Of course that flexibility doesn't come for free and has an overhead. Another benefit is that data-at-rest can be safely decoded for every Erlang version back to at least release 6. This is quite useful in many situations.

So the path is usually to upgrade the OTP version first and then start using new features once the cluster is upgraded.

In the OTP20 release, a change happened in late RCs of the release because serialization stability was brought into jeopardy. It was reported by RabbitMQ and the serializer was changed so it is properly backwards compatible. Upgrade paths are important.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#202

Post by renaudg »

It looks like they have built an interesting, robust and scalable system which is perfectly tailored to their needs.

If one didn't want to build all of that in house though, is there anything they've described here that an off the shelf system like https://socketcluster.io doesn't provide ?

Re: How Discord Scaled Elixir to 5M Concurrent Users

#203

Post by agentgt »

pencilhappen wrote: post #62
Cieplak wrote: post #20

I know that the JVM is a modern marvel of software engineering, so I'm always surprised when my Erlang apps consume less than 10MB of RAM, start up nearly instantaneously, respond to HTTP requests in less than 10ms and run forever, while my Java apps take 2 minutes to start up, have several hundred millisecond HTTP response latency and horde memory. Granted, it's more an issue with Spring than with Java, and Parallel…

> Parallel Universe's Quasar is basically OTP for Java, so I know logically that Java is basically a superset of Erlang at this point It's not really a superset. The JVM has a single global heap, whereas BEAM has per-process heaps. Global state is the enemy of concurrency, and the best GC algorithm is one you don't need to run at all. The "several hundred millisecond HTTP response latency and horde (sic) memory" prob…

I agree. There is some work to go off heap with Java and supposedly you can get incredibly results. Unfortunately the libraries that seem to help doing this are either proprietary are not very well support or stalled.

Of course I think part of the problem is most people in Java land just don't have the 5m concurrency requirements so it doesn't get the love it probably should.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#204

Post by janekm »

josevalim wrote: post #67
janekm wrote: post #46

Earlier quoted context omitted.

I read it a little differently. The whole article is about how Erlang/Elixir fails at its core reason for existence (fast message passing between distributed processes) and all the complicated work-arounds they had to implement to avoid actually using this core feature of Erlang.

People tend to forget that scalability is not a binary property. You always scale up to some users, up to some architecture, up to some amount of nodes. There is no system that will scale to infinity without requiring developer intervention once business needs and application patterns start to settle in. Distributed Erlang/Elixir has known limitations . For example, the network is fully meshed, which gives you about…

Thanks a lot for the very insightful reply. I'm learning a lot about Erlang from the responses, and a lot of respect for the community as well. Not bad for an on reflection somewhat inflammatory comment :)

Re: How Discord Scaled Elixir to 5M Concurrent Users

#205

Post by censhin »

jlouis wrote: post #198
censhin wrote: post #129

Earlier quoted context omitted.

The challenge is going to be adoption, plain and simple. I worked two years on an Erlang project, and most of our frustration came from the lack of community to lean on when we needed help understanding best practices. Also, when you find only one library to solve your problem that has been abandoned for at least two years that is never fun. Few projects actually need the level of scalability that Erlang brings, and…

The key benefit of Erlang isn't the scalability in my experience. It is resilience. Once your system is deployed, it tend to run with 0 fatalities for months on end. We have systems where we've had 0 maintenance days on them for months and even years. In turn, you can focus your development effort on new systems, new features, and scaling existing infrastructure since you are not bogged down in maintenance mode. Even…

I agree with what you said. What I was really trying to get across was the need for a real community and wider adoption.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#206

Post by majidazimi »

jlouis wrote: post #197
majidazimi wrote: post #175

Earlier quoted context omitted.

I did also mention that for fast time to market it is indeed a good idea to use already available tools. By the way writing a distribution layer is not that difficult. Many companies/startups with scalable back-ends that are not using Erlang/OTP have already done it. The point is scalable software requires knowledge and Erlang/OTP is not going to magically solve it. But it seems many fans are trying to promote it suc…

Erlang (and by extension Elixir) definitely provides a set of tools which are good for building highly concurrent distributed systems. And your systems are likely to have few errors as well as being resilient, if you know what you are doing. But indeed, you need to know your tools like every other part of computer science. Storing 9 billion elements in an array, doing linear search and then complaining your linear se…

Consider for some reason (either technical or political), this company decides to migrate their web socket servers to Akka/Rust/Go/NodeJS. Integrating these new servers into their core cluster is going to be deadly painful. They rely heavily on Erlang/OTP internal clustering. This is not even considered as a challenge in distributed systems but still really painful to implement because of their design decision. This is what they did wrong. Clustering and message routing part of the application should be technology agnostic.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#208

Post by lostcolony »

jlouis wrote: post #198
censhin wrote: post #129

Earlier quoted context omitted.

The challenge is going to be adoption, plain and simple. I worked two years on an Erlang project, and most of our frustration came from the lack of community to lean on when we needed help understanding best practices. Also, when you find only one library to solve your problem that has been abandoned for at least two years that is never fun. Few projects actually need the level of scalability that Erlang brings, and…

The key benefit of Erlang isn't the scalability in my experience. It is resilience. Once your system is deployed, it tend to run with 0 fatalities for months on end. We have systems where we've had 0 maintenance days on them for months and even years. In turn, you can focus your development effort on new systems, new features, and scaling existing infrastructure since you are not bogged down in maintenance mode. Even…

For anyone reading this, I worked in a shop where we built an Erlang system, choosing the language specifically for the high uptimes we needed. We nailed it so well that IT proactively reboots the system just because they don't trust that it's still working properly when it runs without issue for months on end. Done right, Erlang can be insanely resilient.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#209

Post by andruby »

tmat wrote: post #111
mahyarm wrote: post #36

Earlier quoted context omitted.

They could side step to corporate slack on-prem chat style solutions and be a pretty strong contender. They were scaling huge chat rooms and such way better than slack was a year ago. But they seem to refuse suggestions to do so continuously, so they seem to have some business plan somewhere.

I discussed moving my companies chat to discord because its a better experience than Slack in every way. They wouldn't go for it because it's marketing is so gamer oriented and wouldn't look professional when we invite clients onboard :(

I have the same problem. It would be great if they released the same product without the gaming brand. I would use that for all my personal and professional chats.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#210

Post by andruby »

Arathorn wrote: post #193
jakebasile wrote: post #39

Earlier quoted context omitted.

If there was an open source and federated equivalent to the features Discord provides I'd use it. There is no such product. Matrix is interesting, but the experience is no where near as polished as Discord and friends and that matters for mass adoption.

We are working as hard as we can on polishing Matrix. Making it mass user friendly is simply our #1 priority. Folks can help directly (beyond PRs, bug reports and feature requests) by donating at https://patreon.com/matrixdotorg .

Maybe Matrix can learn from Discord's technology choices? :)
Post reply on HN