Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Achieving 100k connections per second with Elixir

stressgrid.com

Post reply on HN
Print view View original

21–30 of 117 posts

Re: Achieving 100k connections per second with Elixir

#21

Post by dzik »

lstodd wrote: post #6
zambal wrote: post #4

Earlier quoted context omitted.

We used Ubuntu 18.04 with the 4.15.0-1031-aws kernel, with sysctld overrides seen in our /etc/sysctl.d/10-dummy.conf. We used Erlang 21.2.6-1 on a 36-core c5.9xlarge instance. To run this test, we used Stressgrid with twenty c5.xlarge generators.

omg. 100K/sec was achieved by yours truly 10 years ago on a contemporary xeon with nothing but nginx and python2.6 - gevent patched to not copy the stack, just switch it. (EDIT: and also a FIFO I/O scheduler) Why does this require 36 cores today??

Would you mind sharing the details? (URL maybe)

I think limiting factor might be not number of cores and outside of erl scope, that is eth card they used, network infrastructure, etc. Even Elixir could be something that impacts the tests.

Re: Achieving 100k connections per second with Elixir

#22

Post by dzik »

csisnett wrote: post #17
dzik wrote: post #5

This article is quite good, especially part about bottleneck caused by single supervisor in ranch. However I have to say that title is a bit misleading because all of this has nothing to do with Elixir, it's all about Linux kernel and Erlang, cowboy and ranch are written in Erlang. Having said that, I will add that I think it is good to have Elixir.

"is a bit misleading because all of this has nothing to do with Elixir" Stressgrid is written in elixir though, https://gitlab.com/stressgrid/stressgrid

Point taken and I am already looking at stressgrid, "millions of users" is definitely a selling point to me. It is actually quite hard to generate enough and correct traffic to stress test large distributed systems.

Re: Achieving 100k connections per second with Elixir

#23

Post by makkesk8 »

Even if connections per second can be a magnitude or two lower than requests per second this result is still quite off by today's alternative.

14 core machine comparing .net core with other top webservers: https://www.ageofascent.com/2019/02/04/asp-net-core-saturati...

Re: Achieving 100k connections per second with Elixir

#24

Post by dzik »

muststopmyths wrote: post #20

>Finally, the connections per second rate reaches 99k, with network latency and available CPU resources contributing to the next bottleneck. Can someone educate me on what they might talking about here ? CPU is ~45% in their final graph. I don't know what network latency means in this context though. Roundtrip for a TCP handshake ? That seems unlikely.

It means CPU is not saturated, so it is not the bottleneck, which means it is likely not enough Erlang processes have been started.

Re: Achieving 100k connections per second with Elixir

#25

Post by benfolred »

lstodd wrote: post #6
zambal wrote: post #4

Earlier quoted context omitted.

We used Ubuntu 18.04 with the 4.15.0-1031-aws kernel, with sysctld overrides seen in our /etc/sysctl.d/10-dummy.conf. We used Erlang 21.2.6-1 on a 36-core c5.9xlarge instance. To run this test, we used Stressgrid with twenty c5.xlarge generators.

omg. 100K/sec was achieved by yours truly 10 years ago on a contemporary xeon with nothing but nginx and python2.6 - gevent patched to not copy the stack, just switch it. (EDIT: and also a FIFO I/O scheduler) Why does this require 36 cores today??

You are comparing apples and oranges.

They are purposely holding the connections around for 1+10%seconds. So first of all, it means that, for a rate of 100k conn/s, they are going to have around 200k open connections after a second. This already imposes a different profile than 100k single request connections per second.

You are also assuming that they need 36 cores to achieve 100k connections per second, which is likely not the case since they quickly moved the bottleneck to the OS. I am assuming they have other requirements that force them to run on such a large machine and they want to make sure they are not running into any single-core bottlenecks (and having a large amount of cores makes it much easier to spot those).

Re: Achieving 100k connections per second with Elixir

#26

Post by Qwertystop »

muststopmyths wrote: post #20

>Finally, the connections per second rate reaches 99k, with network latency and available CPU resources contributing to the next bottleneck. Can someone educate me on what they might talking about here ? CPU is ~45% in their final graph. I don't know what network latency means in this context though. Roundtrip for a TCP handshake ? That seems unlikely.

The CPU graph peaks near 97% (teal line) at the time when connections-per-second are highest. Are you looking at the red? That's the version without the two patches.

Re: Achieving 100k connections per second with Elixir

#28

Post by lstodd »

dzik wrote: post #21
lstodd wrote: post #6

Earlier quoted context omitted.

omg. 100K/sec was achieved by yours truly 10 years ago on a contemporary xeon with nothing but nginx and python2.6 - gevent patched to not copy the stack, just switch it. (EDIT: and also a FIFO I/O scheduler) Why does this require 36 cores today??

Would you mind sharing the details? (URL maybe) I think limiting factor might be not number of cores and outside of erl scope, that is eth card they used, network infrastructure, etc. Even Elixir could be something that impacts the tests.

There is no url summing the details unfortunately.

The work in some unknown state is at https://code.google.com/archive/p/coev/

Without the business logic (which was in django IIRC) and deployment details, obviously. Very outdated and some later patches might be missing. No one was interested, you see.

I'd be surprised if there were problems with network, and if there were, that should have been obvious in the metrics.

Maybe the metrics were inadequate

Re: Achieving 100k connections per second with Elixir

#29

Post by Thaxll »

lstodd wrote: post #6
zambal wrote: post #4

Earlier quoted context omitted.

We used Ubuntu 18.04 with the 4.15.0-1031-aws kernel, with sysctld overrides seen in our /etc/sysctl.d/10-dummy.conf. We used Erlang 21.2.6-1 on a 36-core c5.9xlarge instance. To run this test, we used Stressgrid with twenty c5.xlarge generators.

omg. 100K/sec was achieved by yours truly 10 years ago on a contemporary xeon with nothing but nginx and python2.6 - gevent patched to not copy the stack, just switch it. (EDIT: and also a FIFO I/O scheduler) Why does this require 36 cores today??

I highly doubt you were able to do 100k connections/sec 10 years ago with the same hardware, you must be confused between requests/sec and connections/sec very different things.

Re: Achieving 100k connections per second with Elixir

#30

Post by sergiotapia »

makkesk8 wrote: post #23

Even if connections per second can be a magnitude or two lower than requests per second this result is still quite off by today's alternative. 14 core machine comparing .net core with other top webservers: https://www.ageofascent.com/2019/02/04/asp-net-core-saturati...

That's really exciting! As someone who dropped out of .NET entirely around the time ASP.Net MVC2 came out, where do you recommend I start looking into aspnet core / .net core? Do you still write core .net in visual studio? or can you use vscode?
Post reply on HN