Live data from Hacker News

How Discord Scaled Elixir to 5M Concurrent Users

blog.discordapp.com

121–130 of 260 posts

Re: How Discord Scaled Elixir to 5M Concurrent Users

#121
post #10

According to Wikipedia, Discord's initial release was March 2015. Elixir hit 1.0 in September 2014 [0]. That's impressively early for adoption of a language for prototyping and for production. [0] https://github.com/elixir-lang/elixir/releases/tag/v1.0.0

If I remember right, they were using Erlang at the beginning, and moved slowly to Elixir as they got more comfortable, and the ecosystem built up around them.

We started with Elixir, some of our engineers have used Erlang prior though.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#122

Earlier quoted context omitted.

Just look for Erlang tools, there are loads available. The only problem with Elixir right now is sometimes people forget to look at Erlang for already solved problems/tooling/etc. Erlang/BEAM has been alive and well in some very serious mission critical applications for 20 years. Just look at how telecoms used it (its origin).

We use Elixir at work, and Erlang tools have a bit of a discovery problem. You can peek at download counts on Hex, if the author uses it, but that doesn't take account CI systems and such that inflate DL counts, or GitHub only repos that are not on Hex (see things like e.g. shell history prior to OTP20). That said, when there is no Elixir tool to do what I want, the very next thing I do is look for an Erlang tool, an…

Download count is a weak metric for evaluating the quality of a library. I don't think it's a good idea to skip the few minutes it takes to read parts of the documentation and critical parts of the code.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#123

Earlier quoted context omitted.

Just look for Erlang tools, there are loads available. The only problem with Elixir right now is sometimes people forget to look at Erlang for already solved problems/tooling/etc. Erlang/BEAM has been alive and well in some very serious mission critical applications for 20 years. Just look at how telecoms used it (its origin).

I don't think that's the problem for metrics. Exometer is the go-to Erlang library, but its dependencies are a nightmare (e.g. https://github.com/Feuerlabs/exometer/issues/154 or https://github.com/Feuerlabs/exometer#dependency-management ), it's essentially unmaintained, and integrating it with Ecto or Phoenix is roll-your-own. I think there is very much a need for a more modern library that plays nicely with Hex.

> I think there is very much a need for a more modern library [...]

Building a solid standalone monitoring/metrics package is apparently next up on the "todo list" for Chris McCord (creator of Phoenix): https://www.youtube.com/watch?v=pfFpIjFOL-I

Re: How Discord Scaled Elixir to 5M Concurrent Users

#124

Thanks for putting this writeup together! I use Elixir and Erlang every day at work, and the Discord blog has been incredibly useful in terms of pointing me towards the right tooling when I run into a weird performance bottleneck. FastGlobal in particular looks like it nicely solves a problem I've manually had to work around in the past. I'll probably be pulling that into our codebase soon.

Note that Erlang 20 may have solved the problem that FastGlobal tries to fix (that of not copying large amounts of data unnecessarily)

[deleted]

Re: How Discord Scaled Elixir to 5M Concurrent Users

#125
post #38

Earlier quoted context omitted.

> I see a bright future for Elixir, and a breath of fresh air for Erlang. Is Erlang really that much of a barrier? Erlang was a touch odd, but I didn't find the language itself that mind-bending. Wrapping my head around the proper way to structure things and the proper use of the OTP libraries was much more time-consuming. That level of architectural thought doesn't magically go away because you changed the language.…

Unfortunately, a lot of people appear to hate Erlang's syntax to the point that it's a barrier to adoption - or Elixir never would've picked up at all. I don't understand it myself, but there we have it.

I've seen this argument about the syntax countless times and during my first looks at Erlang code I couldn't understand much of it, but after sitting down and deciding to learn it it's an extremely simple syntax. The hard/long thing to fully understand is truly OTP.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#127

Earlier quoted context omitted.

We use Elixir at work, and Erlang tools have a bit of a discovery problem. You can peek at download counts on Hex, if the author uses it, but that doesn't take account CI systems and such that inflate DL counts, or GitHub only repos that are not on Hex (see things like e.g. shell history prior to OTP20). That said, when there is no Elixir tool to do what I want, the very next thing I do is look for an Erlang tool, an…

Download count is a weak metric for evaluating the quality of a library. I don't think it's a good idea to skip the few minutes it takes to read parts of the documentation and critical parts of the code.

I agree completely. I never said that it should be used for evaluation or that you shouldn't read the code or docs.

But then the question is... how do you find high quality libraries without something like download count? The issue is signaling good libraries, with poor discovery. I subscribe to many "libraries worth looking at" mailing lists, the awesome lists, etc. but that is still a pretty coarse net. Searching for libraries on GitHub if you don't know the name already or don't really know what you're specifically looking for is an even worse experience.

Combine that with relatively poor fuzzy search on Hex, where even if you DO know what you're looking for, it leads to a very subpar experience. For example, "e-commerce" leads to different results than "commerce". [0][1] These then exclude [2], which is an ecommerce library available on Hex without that keyword.

[0] https://hex.pm/packages?_utf8=%E2%9C%93&search=ecommerce&sor...

[1] https://hex.pm/packages?search=commerce&sort=downloads

[2] https://hex.pm/packages/ryal_core

Re: How Discord Scaled Elixir to 5M Concurrent Users

#128
Hi community, Let me share my experience with you. I'm a hardcore Rails guy and I've been advocating and teaching Rails to the community for years.

My workflow for trying out a new language involves using the language for a small side project and gradually would try to scale it up. So, here's my summary, my experience of all the languages so far:

Scala - It's a vast academic language (official book is with ~700 pages) with multiple ways of doing things and it's attractiveness for me was the JVM. It's proven, robust and highly scalable. However, the language was not quite easy to understand and the frameworks that I've tried (Play 2, Lift) weren't as easy to transition to, for a Rails developer like me.

Nevertheless, I did build a simple calendar application, but it took me 2 months to learn the language and build it.

GoLang - This was my next bet, although I didn't give up on Scala completely (I know it has its uses), I wanted something simple. I used Go and had the same experience as I had when I used C++. It's a fine language, but, for a simple language, I had to fight a lot with configuration to get it working for me - (For example, it has this crazy concept of GOPATH where your project should reside and if your project isn't there it'll keep complaining). Nevertheless, I build my own (simple) Rails clone in GO and realized this isn't what I was looking for. It took my about a month to conquer the language and build my (simple) side project.

Elixir - Finally, I heard of Elixir on multiple HN Rails release threads and decided to give it a go. I started off with Phoenix. The transition was definitely wayy smoother from Rails, especially considering the founding member of this language was a Rails dev. himself (the author of "devise" gem). At first some concepts seemed different (like piping), but once I got used to it, for me there was no looking back.

All was fine until they released Phoenix 1.3, where they introduced the concept of contexts and (re) introduced Umbrella applications. Basically they encourage you to break your application into smaller applications by business function (similar to microservices) except that you can do this however you like (unopinionated). For example, I broke down my application by business units (Finance, Marketing, etc.). This forced me to re-think my application in a way I never would have thought and by this time I had finished reading all 3 popular books on this topic (Domain Driven Design). I loved how the fact that Elixir's design choices are really well suited for DDD. If you're new to DDD I suggest you try giving it a shot, it really can force you to re-think the way you develop software.

By the end of two weeks after being introduced to Elixir, I picked up the language. In a month and a half, I built a complete Salesforce clone just working on the weekends. And this includes even the UI. And I love how my application is always blazing fast, picks up errors even before it compiles and warns me if I'm no using a variable I defined somewhere.

P.S there IS a small learning curve involved if you're starting out fresh:

1) IF you're used to the Rails asset pipeline, you'll need to learn some new tools like Brunch / Webpack / etc. 2) Understand about contexts & DDD (optional) if you want to better architect your application. 3) There is no return statement in Elixir!

As a Ruby developer, here are my thoughts:

1. So, will I be developing with Rails again? Probably yes, for simpler applications / API servers. 2. Is Ruby dying? No. In fact, I can't wait for Ruby 3.

Some drawbacks of Elixir: 1. Relatively new, so sometimes you'll be on your own and that's okay. 2. Fewer libraries as compared to the Ruby eco-system. But you can easily write your own. 3. Fewer developers, but should be fairly to onboard Ruby developers.

Cheers.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#129
post #19

This writeup make me even more convinced of Elixir becoming one of the large players when it comes to hugely scaling applications. If there is one thing I truly love about Elixir, it is the easiness of getting started, while standing on the shoulders of a giant that is the Erlang VM. You can start by building a simple, not very demanding application with it, yet once you hit a large scale, there is plenty of battle-p…

The challenge is going to be adoption, plain and simple. I worked two years on an Erlang project, and most of our frustration came from the lack of community to lean on when we needed help understanding best practices. Also, when you find only one library to solve your problem that has been abandoned for at least two years that is never fun.

Few projects actually need the level of scalability that Erlang brings, and the cost of building and running an Erlang project as an enterprise application is non-trivial. People need to understand this if they want Elixir to succeed where Erlang hasn't.

Post reply on HN