Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Balancer Battle – Load testing HAproxy, Nginx and HTTP-Proxy's WebSocket support

github.com

Post reply on HN
Print view View original

21–30 of 40 posts

Re: Balancer Battle – Load testing HAproxy, Nginx and HTTP-Proxy's WebSocket support

#21

Post by TallGuyShort »

ominous_prime wrote: post #18

As a simple reverse proxy for small setups, there is almost no difference between the two, especially when running on a VM. You do miss many of the advanced balancing features in haproxy, but again, this config was a basic reverse proxy, not really load-balancing anything. I haven't worked on these in a couple years, but on real hardware, haproxy could push much more bandwidth. We could saturate 10Gb ethernet fairly…

+1 to your second paragraph. I've previously carried out what I considered a pretty fair benchmark: real hardware, realistic server setup, well-informed configuration of both nginx and haproxy, etc... HAProxy had significantly bettter performance in my results. Not questioning their results, just surprised.

Re: Balancer Battle – Load testing HAproxy, Nginx and HTTP-Proxy's WebSocket support

#23

Post by V1 »

thrownaway2424 wrote: post #22

Is it just me, or are all these latency numbers terrible? For a local echo server I would expect mean latency at or below 1ms.

It's not a `local` server. The proxy, websocket server and the benchmark utility are all on different servers.

Re: Balancer Battle – Load testing HAproxy, Nginx and HTTP-Proxy's WebSocket support

#24

Post by fywacro »

devicenull wrote: post #9

Why do people always benchmark on virtual machines running on someone else's server, and expect meaningful results?

These benchmarks DO represent meaningful results.

Some benchmarks are designed to measure absolute performance, in order to answer questions like "How many servers should we buy, if we expect to handle X hits/second?" or "What's the limit of our app stack's scalability, in hits/second?" But... the O.P is NOT benchmarking for absolute performance.

But the O.P. here is benchmarking toward a different purpose: comparing relative performance. These tests are designed to answer a totally different kind of question: "Which of these app stacks performs best, given the same hardware budget for each stack?" or "How many extra servers do we need to buy, if we want to use Apache/mod_php instead of Nginx/php-fpm?"

With relative-performance benchmarks, we have to assume that it's valid to extrapolate from small servers to large, and from one server to many. That is, if Ngnix beats Apache by 1000% on a lonely 500MHz Pentium 3 box, what can we predict about NGinx vs. Apache performance on a dozen-strong cluster of quad-core, dual-socket 3.6GHz machines? In more general terms: How well does each application "scale up/out, horizontally"?

The answer depends on the application type and software architecture. For example, most modern web servers are multi-thread/process apps, with minimal shared state in between. Also, modern web stacks generally push cross-request state into a separate datastore layer (if any). As a result, modern web apps tend to scale up linearly, to the performance limits of the datastore layer. Until your database becomes a bottleneck, you can expect that 2x web servers == 2x hits/second.

Re: Balancer Battle – Load testing HAproxy, Nginx and HTTP-Proxy's WebSocket support

#25

Post by thrownaway2424 »

V1 wrote: post #23
thrownaway2424 wrote: post #22

Is it just me, or are all these latency numbers terrible? For a local echo server I would expect mean latency at or below 1ms.

It's not a `local` server. The proxy, websocket server and the benchmark utility are all on different servers.

I meant "local" as in "not on the other side of a continent". Here's me pinging a nearby DNS server that's 40 miles away from my desk:

  % ping -c 10 -A 4.2.2.1
  PING 4.2.2.1 (4.2.2.1) 56(84) bytes of data.
  64 bytes from 4.2.2.1: icmp_req=1 ttl=56 time=3.24 ms
  64 bytes from 4.2.2.1: icmp_req=2 ttl=56 time=2.88 ms
  64 bytes from 4.2.2.1: icmp_req=3 ttl=56 time=2.95 ms
  64 bytes from 4.2.2.1: icmp_req=4 ttl=56 time=2.90 ms
  64 bytes from 4.2.2.1: icmp_req=5 ttl=56 time=2.95 ms
  64 bytes from 4.2.2.1: icmp_req=6 ttl=56 time=2.91 ms
  64 bytes from 4.2.2.1: icmp_req=7 ttl=56 time=2.90 ms
  64 bytes from 4.2.2.1: icmp_req=8 ttl=56 time=2.87 ms
  64 bytes from 4.2.2.1: icmp_req=9 ttl=56 time=2.94 ms
  64 bytes from 4.2.2.1: icmp_req=10 ttl=56 time=2.94 ms

  --- 4.2.2.1 ping statistics ---
  10 packets transmitted, 10 received, 0% packet loss, time 1806ms
  rtt min/avg/max/mdev = 2.875/2.952/3.247/0.112 ms, ipg/ewma 200.705/3.019 ms
I can't imagine why a local websocket echo service should be 5x slower that this.

Re: Balancer Battle – Load testing HAproxy, Nginx and HTTP-Proxy's WebSocket support

#26

Post by kalmar »

devicenull wrote: post #12
V1 wrote: post #10

Earlier quoted context omitted.

Because we people obviously don't have a datacenter in our own basement. And the common mistake people make when benchmarking is running the servers on their own machine and then use the same machine to benchmark the server it's running. You need to have multiple (powerful) machines for this. And also, spinning up machines in the cloud is quite easy to do and allows people to reproduce the same test results because y…

You can reproduce them? So you can match the loads running on all the other VMs on the same host then?

You could use cluster compute instances which use hardware virtualisation, and I think gives you the whole machine. With spot pricing, you could run them for about 21¢ an hour.

Re: Balancer Battle – Load testing HAproxy, Nginx and HTTP-Proxy's WebSocket support

#28

Post by devicenull »

fywacro wrote: post #24
devicenull wrote: post #9

Why do people always benchmark on virtual machines running on someone else's server, and expect meaningful results?

These benchmarks DO represent meaningful results. Some benchmarks are designed to measure absolute performance, in order to answer questions like "How many servers should we buy, if we expect to handle X hits/second?" or "What's the limit of our app stack's scalability, in hits/second?" But... the O.P is NOT benchmarking for absolute performance. But the O.P. here is benchmarking toward a different purpose: comparing…

He's benchmarking relative performance, but we don't know what else is running on the host machine during the benchmark. What if one of the other users on the host was running a very cpu/network intensive process during only one of his test runs?

Re: Balancer Battle – Load testing HAproxy, Nginx and HTTP-Proxy's WebSocket support

#29

Post by V1 »

TallGuyShort wrote: post #21
ominous_prime wrote: post #18

As a simple reverse proxy for small setups, there is almost no difference between the two, especially when running on a VM. You do miss many of the advanced balancing features in haproxy, but again, this config was a basic reverse proxy, not really load-balancing anything. I haven't worked on these in a couple years, but on real hardware, haproxy could push much more bandwidth. We could saturate 10Gb ethernet fairly…

+1 to your second paragraph. I've previously carried out what I considered a pretty fair benchmark: real hardware, realistic server setup, well-informed configuration of both nginx and haproxy, etc... HAProxy had significantly bettter performance in my results. Not questioning their results, just surprised.

I'll be more then happy to see if my configurations can be tweaked. And I was surprised as well, but it seems that it always boils down to the configuration files a small change can sometimes yield great results as seen with a specifying ciphers for SSL.

Re: Balancer Battle – Load testing HAproxy, Nginx and HTTP-Proxy's WebSocket support

#30

Post by ominous_prime »

V1 wrote: post #29
TallGuyShort wrote: post #21

Earlier quoted context omitted.

+1 to your second paragraph. I've previously carried out what I considered a pretty fair benchmark: real hardware, realistic server setup, well-informed configuration of both nginx and haproxy, etc... HAProxy had significantly bettter performance in my results. Not questioning their results, just surprised.

I'll be more then happy to see if my configurations can be tweaked. And I was surprised as well, but it seems that it always boils down to the configuration files a small change can sometimes yield great results as seen with a specifying ciphers for SSL.

Like I said, running as a simple reverse-proxy, they should be pretty close. The added latency is really the differentiator when going to a single backend, and that's about all you're measuring. You're control itself is still Now add dozens of backend servers, and start throwing a couple orders of magnitude more traffic at it, and the differences will really start to show.

Please note: I'm not saying your numbers aren't useful. Many people will be using a setup like this, and would like to know what the differences would be, if any.

Post reply on HN