Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wordpress.com serves 70k requests/second using Nginx

highscalability.com

Post reply on HN
Print view View original

41–50 of 97 posts

Re: Wordpress.com serves 70k requests/second using Nginx

#41

Post by jacques_chester »

ck2 wrote: post #38
jacques_chester wrote: post #21

I've been administering Wordpress blogs for almost a decade now; hosting for ... 5? 6 years? I'm not sure any more. Anyhow. I started with Apache. Apache probably works fine if you load the mpm-oh-you-didnt-know-about-mpm-so-sorry module. My next stop was lighttpd. A fine piece of software, with a persistent bug that caused it to drop into a 500 state if FastCGI instances were ever unavailable. So now I'm on Nginx. A…

I have uptime of weeks with APC opcode cache without problems on very heavy loaded servers. You can also run a script to watch for PHP/APC segfaults and just restart the the cache. I cannot imagine running PHP without an opcode cache, you are losing a speedup of 300% to 500%

> You can also run a script to watch for PHP/APC segfaults and just restart the the cache.

... wat

> I cannot imagine running PHP without an opcode cache, you are losing a speedup of 300% to 500%

That's just not what my profiling showed.

Re: Wordpress.com serves 70k requests/second using Nginx

#42

Post by seldo »

ww520 wrote: post #34
HyprMusic wrote: post #9

Earlier quoted context omitted.

According to the article they have 2 thousand servers. These are Dual Xeon 5620 4 core CPUs with hyper-threading. So that averages out to: 70000 / (4 * 2000) = 8.75 requests per second per CPU. That seems like quite a low number, I would presume that the load would be shared amongst less servers than this and the others would be used for replication/redundancy.

2000 servers? My god. Aren't most blog posts just static pages or generated static pages? A good html cache setup should help a lot.

They can't possibly mean 2000 of the beefy servers described as being the load-balancers, or it wouldn't be impressive at all. But it's very frustrating that the article gives the specs of the load balancers without saying how many of them they need.

Re: Wordpress.com serves 70k requests/second using Nginx

#43

Post by seldo »

vhf wrote: post #18
HyprMusic wrote: post #9

Earlier quoted context omitted.

According to the article they have 2 thousand servers. These are Dual Xeon 5620 4 core CPUs with hyper-threading. So that averages out to: 70000 / (4 * 2000) = 8.75 requests per second per CPU. That seems like quite a low number, I would presume that the load would be shared amongst less servers than this and the others would be used for replication/redundancy.

It says 12 datacenters distributed around the globe. I think we can safely assume each datacenter holds a dedicated load-balancing server. Let's assume they have only one of these at each datacenter, it gives us 70000 / (4 * 12) = 1460 requests per second per CPU, which is in line with reliably handling over 10,000 request per second of live traffic to WordPress applications from a single server. and In April 2008 Au…

It's definitely not one load balancer, but nor is it 70krps distributed across 2000 servers. I also think having only one per datacenter would be a bit risky -- I'd have at least 2. It's very frustrating that the article doesn't make clear how many load balancers they run.

Re: Wordpress.com serves 70k requests/second using Nginx

#44

Post by barsxl »

mstank wrote: post #37

Article didn't mention that they also use a CDN (EdgeCast). Helps make the 70k requests/second a lot more bearable.

We serve about 150k req/sec from the CDN. That (obviously) isn't included in the 70k number from the article. The 70k is only what we are serving from the origin.

Re: Wordpress.com serves 70k requests/second using Nginx

#45

Post by barsxl »

gregcmartin wrote: post #12

I was one of the core engineers @ layeredtech who managed the servers and HA for wordpress.com when they launched in 2006. If I remember correctly we were using DNS round robin and haproxy -> apache -> mysql all on freebsd systems wow have things come a long ways since then also it's incredible the sustained growth of Wordpress after all this time. good memories... congrats Matt on all your success.

WordPress.com hasn't run on FreeBSD since the TextDrive days which was way before Layered Tech, before me and before WordPress.com was open to the public. We are 100% Debian today, but have used Ubuntu in the past. We never really used HAProxy either. I posted about out load balancer choices back in 2008 - http://barry.wordpress.com/2008/04/28/load-balancer-update/

Re: Wordpress.com serves 70k requests/second using Nginx

#46

Post by barsxl »

sowhata wrote: post #6

And how many requests/sec per CPU? They have like 300 servers and who knows how many cores. What kind title is this? The target High Scalability reader must be presumed to be a complete fool.

As with any tiered web application, the requests aren't evenly distributed across all the servers :) Most of our main Nginx proxies serve about 5k req/sec and have about 50k established connections. They are usually 8 cores + HT for a total of 16 "threads". They aren't at 100% utilization and the limiting factor isn't Nginx CPU usage - it's software interrupts generated by the NICs or bandwidth or something else... We have seen single machines serve upwards of 20k req/sec under "real world" conditions before.

Re: Wordpress.com serves 70k requests/second using Nginx

#47

Post by ck2 »

jacques_chester wrote: post #41
ck2 wrote: post #38

Earlier quoted context omitted.

I have uptime of weeks with APC opcode cache without problems on very heavy loaded servers. You can also run a script to watch for PHP/APC segfaults and just restart the the cache. I cannot imagine running PHP without an opcode cache, you are losing a speedup of 300% to 500%

> You can also run a script to watch for PHP/APC segfaults and just restart the the cache. ... wat > I cannot imagine running PHP without an opcode cache, you are losing a speedup of 300% to 500% That's just not what my profiling showed.

You mentioned opcode cache problems - some people complain of segfaults and I am just pointing out you can make it restart if that ever happens. (I've had it trigger three times this entire year so far.)

There is most certainly a serious load reduction when using an opcode cache. It's not just common sense, you easily find a dozen independent benchmarks on the web proving it.

Now if you had the opcode cache running incorrectly you might not see the improvement (ie. some people misconfigure it where the memory is not shared and persistant, and instead gets destroyed as php children are created/removed).

Re: Wordpress.com serves 70k requests/second using Nginx

#48

Post by barsxl »

ck2 wrote: post #2

Those must be cached pages from dozens of servers on the backend. There is no way in heck it's realtime queries. I can tell when authors are in the WP backend on the server just by looking at the server load because it's crippling. The wordpress bottleneck is not nginx vs apache vs whatever, it's the problem of loading hundreds of files for any kind of page render (even to just authenticate for ajax, etc.) and over a…

Many more dynamic pages than you would expect. We do hundreds of thousands of database queries/sec. We do some caching with Batcache - http://wordpress.org/extend/plugins/batcache/ but it's more to handle large spikes in traffic for single pages/blogs. CNN during the US elections, for example.

Re: Wordpress.com serves 70k requests/second using Nginx

#49

Post by ck2 »

nisa wrote: post #20
ck2 wrote: post #2

Those must be cached pages from dozens of servers on the backend. There is no way in heck it's realtime queries. I can tell when authors are in the WP backend on the server just by looking at the server load because it's crippling. The wordpress bottleneck is not nginx vs apache vs whatever, it's the problem of loading hundreds of files for any kind of page render (even to just authenticate for ajax, etc.) and over a…

Wordpress is only reasonable fast if you use APC, memcached and the MySQL query cache. I never liked the caching plugins. If you use nginx just use the fastcgi_cache module and make cookies part of the cache key or omit the caching of pages where the login cookie is set.

APC+mysql cache (not just query but keycache too) is a must, but memcached is pointless unless you have a multi-server website (on a single server APC shared memory is way faster than memcached).

All logged out users should be served a completely cached static page that bypasses PHP entirely, it's way faster than even an opcode cache and way less load.

WP-Super-Cache can serve static pages to logged out users and makes this rather easy to setup. It's practically a must for wordpress.

Re: Wordpress.com serves 70k requests/second using Nginx

#50

Post by barsxl »

smcnally wrote: post #36
jacques_chester wrote: post #21

I've been administering Wordpress blogs for almost a decade now; hosting for ... 5? 6 years? I'm not sure any more. Anyhow. I started with Apache. Apache probably works fine if you load the mpm-oh-you-didnt-know-about-mpm-so-sorry module. My next stop was lighttpd. A fine piece of software, with a persistent bug that caused it to drop into a 500 state if FastCGI instances were ever unavailable. So now I'm on Nginx. A…

What's Percona's role in this environment?

We started using Percona's MySQL builds by default a couple of years ago. There are some nice performance and convenience features not included in the MySQL.com builds. (Un)fortunately we still have quite a few MySQL 4.1 instances happily running and are still using MyISAM across over 360 million tables. Most of that stuff is not running Percona.
Post reply on HN