Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wordpress.com serves 70k requests/second using Nginx

highscalability.com

Post reply on HN
Print view View original

51–60 of 97 posts

Re: Wordpress.com serves 70k requests/second using Nginx

#51

Post by barsxl »

seldo wrote: post #43
vhf wrote: post #18

Earlier quoted context omitted.

It says 12 datacenters distributed around the globe. I think we can safely assume each datacenter holds a dedicated load-balancing server. Let's assume they have only one of these at each datacenter, it gives us 70000 / (4 * 12) = 1460 requests per second per CPU, which is in line with reliably handling over 10,000 request per second of live traffic to WordPress applications from a single server. and In April 2008 Au…

It's definitely not one load balancer, but nor is it 70krps distributed across 2000 servers. I also think having only one per datacenter would be a bit risky -- I'd have at least 2. It's very frustrating that the article doesn't make clear how many load balancers they run.

WordPress.com currently has 12 load balancers per data center. They are HA and used for different subsets of traffic.

Re: Wordpress.com serves 70k requests/second using Nginx

#52

Post by stickhandle »

Two takeaways - (1) The article is bumf wrt the Wordpress.com setup. NGINX plays a role, but the thousands of servers for http, db, memcache, load balancing etc is integral to this discussion (and not mentioned). A typical reader couldn't relate if it did. (2) Directed at a more typical reader, use NGINX for hosting your WP.org blogs, company sites etc. Properly configured, your performance to cost ratio will be terrific. Lastly, on a personal note, it pains me when caching plugins are mentioned as the main part of the solution. No need for these folks. Use the NGINX cache instead and never worry again about a buggy/outdated cache plugin (looking at you W3TC) again. Remove that dependency and you will be happier and possibly better looking.

Re: Wordpress.com serves 70k requests/second using Nginx

#53

Post by barsxl »

cowsaysoink wrote: post #11
HyprMusic wrote: post #9

Earlier quoted context omitted.

According to the article they have 2 thousand servers. These are Dual Xeon 5620 4 core CPUs with hyper-threading. So that averages out to: 70000 / (4 * 2000) = 8.75 requests per second per CPU. That seems like quite a low number, I would presume that the load would be shared amongst less servers than this and the others would be used for replication/redundancy.

I don't think that is right, it says that automattic has 2000 servers but they also run things like gravitar, akismet, and vaultpress. Which makes me think that using that number is completely wrong. (Also these are just the load balancers not the back end, which could be taking up a large amount of that 2000 server count) From the article it seems more like they have ~100, maybe less, for the load balancers which co…

Of the 2000 servers, about 90% of them are running something related to WordPress.com. There are 36 "load balancer" servers in total. I added up the req/sec across those 36 machines and came up with the 70k/sec number in the article. The requests aren't evenly distributed across that subset of machines though, so you can't just divide evenly to figure out a req/sec/CPU rate. I left another comment in this thread that mentions 5k req/sec on a "normal" load balancer and that the limiting factor isn't Nginx CPU usage.

Re: Wordpress.com serves 70k requests/second using Nginx

#54

Post by barsxl »

ck2 wrote: post #38
jacques_chester wrote: post #21

I've been administering Wordpress blogs for almost a decade now; hosting for ... 5? 6 years? I'm not sure any more. Anyhow. I started with Apache. Apache probably works fine if you load the mpm-oh-you-didnt-know-about-mpm-so-sorry module. My next stop was lighttpd. A fine piece of software, with a persistent bug that caused it to drop into a 500 state if FastCGI instances were ever unavailable. So now I'm on Nginx. A…

I have uptime of weeks with APC opcode cache without problems on very heavy loaded servers. You can also run a script to watch for PHP/APC segfaults and just restart the the cache. I cannot imagine running PHP without an opcode cache, you are losing a speedup of 300% to 500%

The majority of our transient bugs on WordPress.com are probably related to APC opcode issues. We try to catch them and handle it gracefully, but it isn't always so easy.

Re: Wordpress.com serves 70k requests/second using Nginx

#55

Post by jacques_chester »

ck2 wrote: post #47
jacques_chester wrote: post #41

Earlier quoted context omitted.

> You can also run a script to watch for PHP/APC segfaults and just restart the the cache. ... wat > I cannot imagine running PHP without an opcode cache, you are losing a speedup of 300% to 500% That's just not what my profiling showed.

You mentioned opcode cache problems - some people complain of segfaults and I am just pointing out you can make it restart if that ever happens. (I've had it trigger three times this entire year so far.) There is most certainly a serious load reduction when using an opcode cache. It's not just common sense, you easily find a dozen independent benchmarks on the web proving it. Now if you had the opcode cache running i…

My point is that loading, parsing and interpreting PHP is rarely the bottleneck in Wordpress.

In terms of microbenchmarks, opcode caches look spectacular. In terms of the larger stack, given the way Wordpress works? Meh, a few percent.

It might be worth it for a big site to shave a few dozen servers off the bill. But the overhead imposed by flakiness is not worth my time.

Re: Wordpress.com serves 70k requests/second using Nginx

#56

Post by jacques_chester »

barsxl wrote: post #54
ck2 wrote: post #38

Earlier quoted context omitted.

I have uptime of weeks with APC opcode cache without problems on very heavy loaded servers. You can also run a script to watch for PHP/APC segfaults and just restart the the cache. I cannot imagine running PHP without an opcode cache, you are losing a speedup of 300% to 500%

The majority of our transient bugs on WordPress.com are probably related to APC opcode issues. We try to catch them and handle it gracefully, but it isn't always so easy.

Out of the opcode caches I've tried, XCache was the most stable. APC was an utter bomb, I wouldn't trust it with burnt hair.

Re: Wordpress.com serves 70k requests/second using Nginx

#57

Post by tlack »

jacques_chester wrote: post #39
mrb wrote: post #35

Earlier quoted context omitted.

My next stop was lighttpd. A fine piece of software, with a persistent bug that caused it to drop into a 500 state if FastCGI instances were ever unavailable. It is not a bug, it is a feature. lighttpd used to wait 60s before checking the backend again. Nowadays the default is 1s. Set disable-time to 0 if you don't like it (it should be the default IMHO.) http://redmine.lighttpd.net/projects/lighttpd/wiki/Docs_ModF..…

The problem was not that lighty gave a 500 error for transient unavailability. It's that it got stuck in 500. I would have to log in every week or so and restart it when this happened.

In case you're wondering whatever happened to this bug, it's still happening! I installed the latest release of Lighttpd a few weeks ago and it happened on one of our sites. So amazing to see the same bug still kicking after 3-4 years..

Re: Wordpress.com serves 70k requests/second using Nginx

#58

Post by mikescar »

jacques_chester wrote: post #56
barsxl wrote: post #54

Earlier quoted context omitted.

The majority of our transient bugs on WordPress.com are probably related to APC opcode issues. We try to catch them and handle it gracefully, but it isn't always so easy.

Out of the opcode caches I've tried, XCache was the most stable. APC was an utter bomb, I wouldn't trust it with burnt hair.

I'm interested to hear what versions of PHP and APC you used.

In my experience, stable APC releases have been very stable, but beta APC can poop out pretty bad, even if the changelog doesn't indicate anything that might affect your application.

I've used stable APC + PHP + Apache releases to do some large things, so not sure why APC is an utter bomb in 2012.

Re: Wordpress.com serves 70k requests/second using Nginx

#59

Post by evansolomon »

jacques_chester wrote: post #56
barsxl wrote: post #54

Earlier quoted context omitted.

The majority of our transient bugs on WordPress.com are probably related to APC opcode issues. We try to catch them and handle it gracefully, but it isn't always so easy.

Out of the opcode caches I've tried, XCache was the most stable. APC was an utter bomb, I wouldn't trust it with burnt hair.

APC is also the most likely by 1000x to be merged into PHP core, so I would rather use/support/fix that.

Re: Wordpress.com serves 70k requests/second using Nginx

#60

Post by jacques_chester »

mikescar wrote: post #58
jacques_chester wrote: post #56

Earlier quoted context omitted.

Out of the opcode caches I've tried, XCache was the most stable. APC was an utter bomb, I wouldn't trust it with burnt hair.

I'm interested to hear what versions of PHP and APC you used. In my experience, stable APC releases have been very stable, but beta APC can poop out pretty bad, even if the changelog doesn't indicate anything that might affect your application. I've used stable APC + PHP + Apache releases to do some large things, so not sure why APC is an utter bomb in 2012.

We're talking ... late 2007 here. I must confess that I didn't keep an engineering diary.
Post reply on HN