Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wordpress.com serves 70k requests/second using Nginx

highscalability.com

Post reply on HN
Print view View original

11–20 of 97 posts

Re: Wordpress.com serves 70k requests/second using Nginx

#11

Post by cowsaysoink »

HyprMusic wrote: post #9
sowhata wrote: post #6

And how many requests/sec per CPU? They have like 300 servers and who knows how many cores. What kind title is this? The target High Scalability reader must be presumed to be a complete fool.

According to the article they have 2 thousand servers. These are Dual Xeon 5620 4 core CPUs with hyper-threading. So that averages out to: 70000 / (4 * 2000) = 8.75 requests per second per CPU. That seems like quite a low number, I would presume that the load would be shared amongst less servers than this and the others would be used for replication/redundancy.

I don't think that is right, it says that automattic has 2000 servers but they also run things like gravitar, akismet, and vaultpress. Which makes me think that using that number is completely wrong. (Also these are just the load balancers not the back end, which could be taking up a large amount of that 2000 server count)

From the article it seems more like they have ~100, maybe less, for the load balancers which comes out to 175 requests/second per cpu which is getting a little more reasonable.

Re: Wordpress.com serves 70k requests/second using Nginx

#12

Post by gregcmartin »

I was one of the core engineers @ layeredtech who managed the servers and HA for wordpress.com when they launched in 2006.

If I remember correctly we were using DNS round robin and haproxy -> apache -> mysql all on freebsd systems wow have things come a long ways since then also it's incredible the sustained growth of Wordpress after all this time. good memories... congrats Matt on all your success.

Re: Wordpress.com serves 70k requests/second using Nginx

#13

Post by ck2 »

TillE wrote: post #10
ck2 wrote: post #7

Earlier quoted context omitted.

Single author mini-blogs without logged in users are super easy to cache. But multi-author blogs with thousands of logged in users is a nightmare with wp. I suspect 90%+ of wp blogs are in the mini-blog category though.

Why would multiple authors make a difference?

Every time an article is created, saved or published, wp uses hundreds of non-cachable queries.

Every time an article is published, it causes the cache to be deleted for not only that article but related pages, which means all those pages have to be rendered again.

For one author, that can be managed. Many authors, the cache is constantly being defeated.

Re: Wordpress.com serves 70k requests/second using Nginx

#14

Post by robmil »

ck2 wrote: post #4
piers wrote: post #3

Earlier quoted context omitted.

Don't forget the version of Wordpress that you download is very different from the version that's on Wordpress.com

I know they replace the database class on wp.com to support replication (I think they even released the code once) but I don't think they do extensive changes otherwise unless you've actually read otherwise.

It's called HyperDB, and they did indeed release it: http://codex.wordpress.org/HyperDB

Re: Wordpress.com serves 70k requests/second using Nginx

#15

Post by [deleted] »

HyprMusic wrote: post #9
sowhata wrote: post #6

And how many requests/sec per CPU? They have like 300 servers and who knows how many cores. What kind title is this? The target High Scalability reader must be presumed to be a complete fool.

According to the article they have 2 thousand servers. These are Dual Xeon 5620 4 core CPUs with hyper-threading. So that averages out to: 70000 / (4 * 2000) = 8.75 requests per second per CPU. That seems like quite a low number, I would presume that the load would be shared amongst less servers than this and the others would be used for replication/redundancy.

[deleted]

Re: Wordpress.com serves 70k requests/second using Nginx

#16

Post by sowhata »

cowsaysoink wrote: post #11
HyprMusic wrote: post #9

Earlier quoted context omitted.

According to the article they have 2 thousand servers. These are Dual Xeon 5620 4 core CPUs with hyper-threading. So that averages out to: 70000 / (4 * 2000) = 8.75 requests per second per CPU. That seems like quite a low number, I would presume that the load would be shared amongst less servers than this and the others would be used for replication/redundancy.

I don't think that is right, it says that automattic has 2000 servers but they also run things like gravitar, akismet, and vaultpress. Which makes me think that using that number is completely wrong. (Also these are just the load balancers not the back end, which could be taking up a large amount of that 2000 server count) From the article it seems more like they have ~100, maybe less, for the load balancers which co…

If this is just about load balancing, then they could probably get better performance from haproxy, right? Anyone should be able to exceed 200 req/s with haproxy. In fact, you should be able to get around double that or more.

The point is the title is useless. It tells you almost nothing. That blog is a joke.

Re: Wordpress.com serves 70k requests/second using Nginx

#17

Post by robmil »

HyprMusic wrote: post #9
sowhata wrote: post #6

And how many requests/sec per CPU? They have like 300 servers and who knows how many cores. What kind title is this? The target High Scalability reader must be presumed to be a complete fool.

According to the article they have 2 thousand servers. These are Dual Xeon 5620 4 core CPUs with hyper-threading. So that averages out to: 70000 / (4 * 2000) = 8.75 requests per second per CPU. That seems like quite a low number, I would presume that the load would be shared amongst less servers than this and the others would be used for replication/redundancy.

They have two thousand servers total; that includes database servers, memcached servers, regular backends, and load balancers.

Peter Westwood gave a talk on WP.com's infrastructure in London in January; I've tried to find slides online but I can't, which is a shame because he went into quite a bit of detail about their nginx/HyperDB/memcached/mogileFS setup.

Re: Wordpress.com serves 70k requests/second using Nginx

#18

Post by vhf »

HyprMusic wrote: post #9
sowhata wrote: post #6

And how many requests/sec per CPU? They have like 300 servers and who knows how many cores. What kind title is this? The target High Scalability reader must be presumed to be a complete fool.

According to the article they have 2 thousand servers. These are Dual Xeon 5620 4 core CPUs with hyper-threading. So that averages out to: 70000 / (4 * 2000) = 8.75 requests per second per CPU. That seems like quite a low number, I would presume that the load would be shared amongst less servers than this and the others would be used for replication/redundancy.

It says 12 datacenters distributed around the globe.

I think we can safely assume each datacenter holds a dedicated load-balancing server. Let's assume they have only one of these at each datacenter, it gives us 70000 / (4 * 12) = 1460 requests per second per CPU, which is in line with reliably handling over 10,000 request per second of live traffic to WordPress applications from a single server.

and

In April 2008 Automattic converted all WordPress.com load balancers from Pound to NGINX. , which points to the fact that they're only talking about load balancers and they have several of them.

This link implies that one NGINX load-balancing instance cannot handle 70k r/s.

Re: Wordpress.com serves 70k requests/second using Nginx

#19

Post by diego »

ck2 wrote: post #13
TillE wrote: post #10

Earlier quoted context omitted.

Why would multiple authors make a difference?

Every time an article is created, saved or published, wp uses hundreds of non-cachable queries. Every time an article is published, it causes the cache to be deleted for not only that article but related pages, which means all those pages have to be rendered again. For one author, that can be managed. Many authors, the cache is constantly being defeated.

What matters is the ratio of reads per write. Blogs with multiple authors have a higher ratio than blogs with single authors.

Re: Wordpress.com serves 70k requests/second using Nginx

#20

Post by nisa »

ck2 wrote: post #2

Those must be cached pages from dozens of servers on the backend. There is no way in heck it's realtime queries. I can tell when authors are in the WP backend on the server just by looking at the server load because it's crippling. The wordpress bottleneck is not nginx vs apache vs whatever, it's the problem of loading hundreds of files for any kind of page render (even to just authenticate for ajax, etc.) and over a…

Wordpress is only reasonable fast if you use APC, memcached and the MySQL query cache. I never liked the caching plugins. If you use nginx just use the fastcgi_cache module and make cookies part of the cache key or omit the caching of pages where the login cookie is set.
Post reply on HN