Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Discord Scaled Elixir to 5M Concurrent Users

blog.discordapp.com

Post reply on HN
Print view View original

131–140 of 260 posts

Re: How Discord Scaled Elixir to 5M Concurrent Users

#131

Post by RexM »

whitepoplar wrote: post #100
josevalim wrote: post #67

Earlier quoted context omitted.

People tend to forget that scalability is not a binary property. You always scale up to some users, up to some architecture, up to some amount of nodes. There is no system that will scale to infinity without requiring developer intervention once business needs and application patterns start to settle in. Distributed Erlang/Elixir has known limitations . For example, the network is fully meshed, which gives you about…

This is a great point! What would you say is the best book to really learn OTP?

Designing for Scalability with Erlang/OTP is pretty good so far.

http://shop.oreilly.com/product/0636920024149.do

Re: How Discord Scaled Elixir to 5M Concurrent Users

#133

Post by rdtsc »

pmarreck wrote: post #78
rdtsc wrote: post #5

Good stuff. Erlang VM FTW! > mochiglobal, a module that exploits a feature of the VM: if Erlang sees a function that always returns the same constant data, it puts that data into a read-only shared heap that processes can access without copying the data There is a nice new OTP 20.0 optimization - now the value doesn't get copied even on message sends on the local node. Jesper L. Andersen (jlouis) talked about it in h…

I would like to subscribe to your newsletter Seriously, though, I might need your services in the future, if you're available. Also, I'm pretty sure the security of your career is pretty guaranteed at this point lol (looking at Indeed data, interest in Elixir has risen 20fold in the past 3 years and the slope of that line is far steeper than all other languages in that space)

I don't have a newsletter. I thought of having a blog for a bunch of stuff like this but I like writing software more than blog posts and I knew it'd abandon it after a while. Maybe there is "turn all your hn posts higher than X upvotes into a blog post series" script somewhere.

However you can subscribe to the Erlang mailing list. That's often a good place to start for tricky or interesting questions you might have:

http://erlang.org/mailman/listinfo/erlang-questions

To contact me directly check my profile.

Cheers!

Re: How Discord Scaled Elixir to 5M Concurrent Users

#134

Post by rdtsc »

lojack wrote: post #75
janekm wrote: post #46

Earlier quoted context omitted.

I read it a little differently. The whole article is about how Erlang/Elixir fails at its core reason for existence (fast message passing between distributed processes) and all the complicated work-arounds they had to implement to avoid actually using this core feature of Erlang.

I read it as "Elixir was a good choice, but it isn't a silver bullet." The fact that they spend most of the article talking about the gotcha's makes their statement about going with Elixir given the opportunity to repeat more favorable in my mind.

For a scalable 5M connected users system no language is a silver bullet. Some tools are just better suites for some jobs. Elixir/Erlang OTP just happen to be suited very well for these kind of jobs.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#135

Post by denisw »

_asummers wrote: post #72
jaequery wrote: post #18

Anyone know if Phoenix/Elixir have something similar to Ruby's bettererror gem? I see Phoenix has a built-in error stack trace page which looks like a clone of bettererror but it doesn't have the real-time console inside of it. Also, I wish they had a ORM like Sequel. These two are really what is holding me back from going full in on Elixir. Anyone can care to comment on this?

Ecto is hands down the best "ORM" I've used. At no point has it ever gotten in the way of us dynamically creating queries on the fly, just following the Ecto internals (not recommended, but totally doable) and has an extremely expressive syntax. The team behind it has been rolling out new features and are extremely responsive to requests on the mailing list. I encourage you to give it a chance. The mental shift is th…

Could you explain which changes to Elixir 1.5 and OTP 20 should result in better Dialyzer error messages? I didn't find anything relevant to that in the respective changelogs.

(For context, Dialyzer is a static analysis tool detecting type errors which is part of the core Erlang distribution.)

Re: How Discord Scaled Elixir to 5M Concurrent Users

#136

Post by rdtsc »

bsder wrote: post #38
iagooar wrote: post #19

This writeup make me even more convinced of Elixir becoming one of the large players when it comes to hugely scaling applications. If there is one thing I truly love about Elixir, it is the easiness of getting started, while standing on the shoulders of a giant that is the Erlang VM. You can start by building a simple, not very demanding application with it, yet once you hit a large scale, there is plenty of battle-p…

> I see a bright future for Elixir, and a breath of fresh air for Erlang. Is Erlang really that much of a barrier? Erlang was a touch odd, but I didn't find the language itself that mind-bending. Wrapping my head around the proper way to structure things and the proper use of the OTP libraries was much more time-consuming. That level of architectural thought doesn't magically go away because you changed the language.…

> Is Erlang really that much of a barrier?

It's not. Not for me at least. I prefer Erlang. Maybe I am strange like that.

I found initially the core concepts are hard - that is using using processes for concurrency, functional (immutable) data structures, functional patterns like recursion instead of for loops, the library ecosystems those are harder things. Those are the same in Elixir as well.

Erlang the language itself also simple. Think a bit like C and C++ if you're familiar. Erlang is like C, the language spec is small. Elixir has additional features which make it more expressive but also more complicated (macros, pipes). It's a bit like C++ having templates and classes. You can do more and with them, but it's a also a bit more to learn. I am exaggerating as Elixir is a lot more elegant and consistent than C++, I am just using the analogy to illustrate the idea of simplicity vs power.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#137

Post by rdtsc »

Cieplak wrote: post #20

I know that the JVM is a modern marvel of software engineering, so I'm always surprised when my Erlang apps consume less than 10MB of RAM, start up nearly instantaneously, respond to HTTP requests in less than 10ms and run forever, while my Java apps take 2 minutes to start up, have several hundred millisecond HTTP response latency and horde memory. Granted, it's more an issue with Spring than with Java, and Parallel…

> Java is basically a superset of Erlang at this point

It's not a superset until it has a non-sharable memory heaps between threads, complete and easy hot code reloading, dynamic tracing (being able to log into a node and update code at will as it the application is running).

The safety and fault tolerance is the #1 advantage Erlang has and that it's hard to get with other frameworks that claim to be Erlang-like. Almost all of them focus on "We have a thread and a queue and we send messages between them so we have 90% of Erlang but faster". Sure they can spawn OS processes to get the same effect or even whole servers but it's more awkward and can only be done so many times before memory or CPU resources are exhausted.

Oh you can think of it another way. There is no point in having a distributed system with 5M concurrently connected users if it crashes 2 or 3 times per day and it has to be restarted and all those users lose their connections. So as the system gets more distributed and more scalable the fault tolerance aspect starts to move to the front alongside speed and performance. And that's just where Erlang starts to shine so to speak.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#138

Post by amorphid »

pmarreck wrote: post #83
Cieplak wrote: post #20

I know that the JVM is a modern marvel of software engineering, so I'm always surprised when my Erlang apps consume less than 10MB of RAM, start up nearly instantaneously, respond to HTTP requests in less than 10ms and run forever, while my Java apps take 2 minutes to start up, have several hundred millisecond HTTP response latency and horde memory. Granted, it's more an issue with Spring than with Java, and Parallel…

> while my Java apps take 2 minutes to start up, have several hundred millisecond HTTP response latency and hoard memory yeah this is pretty much exactly why I turned away from Clojure and Scala and went for Elixir. Also, super ugly Java stacktraces that were not hard to trigger at all. The vast majority of errors in Elixir are wonderfully explanatory.

A lot of the Erlang-y error messages that bubble up into Elixir can get pretty gnarly. Earlier today I accidentally tried doing `Map.get(key, map)` (map should be first arg) in a `&handle_call/1` callback on a GenServer module, and it got ugly fast. I knew what the problem was from experience, but the Erlang-y error can be intimidating to see for the first time.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#139

Post by rdtsc »

jlouis wrote: post #92
rdtsc wrote: post #5

Good stuff. Erlang VM FTW! > mochiglobal, a module that exploits a feature of the VM: if Erlang sees a function that always returns the same constant data, it puts that data into a read-only shared heap that processes can access without copying the data There is a nice new OTP 20.0 optimization - now the value doesn't get copied even on message sends on the local node. Jesper L. Andersen (jlouis) talked about it in h…

It isn't that likely the OTP20 optimization helps here. If the process never sends a message containing the literal value, then there is no benefit in the optimization. What `mochiglobal` and friends are good at is when you have a large set of data (A ring, say) which update rarely, so you can treat it as semi-static data in the system. But then you shouldn't really send that ring data around in the system too much,…

> It isn't that likely the OTP20 optimization helps here

Ah good point. I didn't look at the code much. I was thinking of cases of passing any of those literals in gen_server calls and such and just getting extra performance from upgrading to OTP20 as a side-effect.

Re: How Discord Scaled Elixir to 5M Concurrent Users

#140

Post by eggy »

pmarreck wrote: post #55
bsder wrote: post #38

Earlier quoted context omitted.

> I see a bright future for Elixir, and a breath of fresh air for Erlang. Is Erlang really that much of a barrier? Erlang was a touch odd, but I didn't find the language itself that mind-bending. Wrapping my head around the proper way to structure things and the proper use of the OTP libraries was much more time-consuming. That level of architectural thought doesn't magically go away because you changed the language.…

Syntax is a funny thing. It seems that some developers (perhaps most) care about it (disclaimer: myself included) but there's a large number I've encountered who do not. It seems to be a subjective preferential thing, because most rational discussions I've attempted to have about it result in circling around a drain and coming to no conclusion. I'd say that they're simply blind to it (similar to how a colorblind pers…

I tried translating some of the book "The Handbook of Neuroevolution Through Erlang" into LFE (Lisp Flavored Erlang), and Elxir (somebody's done this already I believe).

Robert Virding, a co-creator of Erlang, creator of LFE, also wrote Luerl - Lua implemented in Erlang.

So if syntax is important you have many choices to use the BEAM VM/OTP in a Ruby-like language (Elixr), a Lisp (LFE) or in Lua (Luerl)!

I am now looking at Robert's videos of a game whose logic is written Lua, but it runs an Erlang process for each ship of thousands he spawns in the demo. Pretty cool. There's even Torchcraft - Luerl and Torch ML for learning AI/ML in Starcraft.

I still prefer LFE because I like Lisp syntax, biased as yourself about such things, however, I still think Elixir macros are not as far reaching as LFE's since they are handled after parsing in Elixir vs. LFE [1], although I wouldn't mind being proved incorrect on this.

  [1]  https://groups.google.com/forum/#!topic/lisp-flavoured-erlang/ensAkzcEDQQ
Post reply on HN