Live data from Hacker News

Wordpress.com serves 70k requests/second using Nginx

highscalability.com

21–30 of 97 posts

Re: Wordpress.com serves 70k requests/second using Nginx

#21
I've been administering Wordpress blogs for almost a decade now; hosting for ... 5? 6 years? I'm not sure any more.

Anyhow. I started with Apache. Apache probably works fine if you load the mpm-oh-you-didnt-know-about-mpm-so-sorry module.

My next stop was lighttpd. A fine piece of software, with a persistent bug that caused it to drop into a 500 state if FastCGI instances were ever unavailable.

So now I'm on Nginx. And with some fancy footwork in the rewrite rules to support WP Super Cache, it can serve static files and gzipped HTML straight off disk.

Unsurprisingly, in this configuration, it runs like the clappers.[1][2]

One thing that always surprises me is seeing configurations where Nginx is merely a proxy or load balancer into a bunch of Apache instances. Are you nuts? Stop doing that. Just use Nginx and FastCGI -- thank me later.

[1] At least, it used to do this, until I updated everything and it silently stopped working the way it used to (which I only discovered while outlining my current config in comments below). I mean, thanks to memcached and closer servers and running Percona on a separate server it's still pretty fast, but this change bugs me and you can bet I'll be fixing it later today after work.

[2] Two optimisations I've dropped are the MySQL query cache and a PHP opcode cache.

The query cache because I don't think it buys me enough versus using memcached (just a gut feel, I've got no hard numbers to back me up) and because it often turns into a contention point in read-heavy MySQL instances. Also, by dropping it I can free up more memory for InnoDB.

The opcode cache because they tend to be flaky. I've had bad experiences with all three major ones. If you ever pop Wordpress up on a profiled PHP instance you find that its time is spent overwhelmingly on waiting for MySQL, concatenating strings and then streaming it out to the webserver, for which proper on-disk/in-memory caching is the answer. Time spent loading and compiling .php is minimal by comparison, so why put up with the hassle?

I don't use varnish because the whole philosophy is to let the OS select what to cache in memory, and ... well I already do that. Plus getting ESI to work on frequently updated objects like "recent comment" widgets is a hairy pain in the arse and I just can't be bothered.

Re: Wordpress.com serves 70k requests/second using Nginx

#22
post #16

Earlier quoted context omitted.

I don't think that is right, it says that automattic has 2000 servers but they also run things like gravitar, akismet, and vaultpress. Which makes me think that using that number is completely wrong. (Also these are just the load balancers not the back end, which could be taking up a large amount of that 2000 server count) From the article it seems more like they have ~100, maybe less, for the load balancers which co…

If this is just about load balancing, then they could probably get better performance from haproxy, right? Anyone should be able to exceed 200 req/s with haproxy. In fact, you should be able to get around double that or more. The point is the title is useless. It tells you almost nothing. That blog is a joke.

Well it is average req/s so peak should be much higher, and they probably can average much more than what they are doing now in case of spikes and other such things.

But yeah the title is pretty much useless.

Re: Wordpress.com serves 70k requests/second using Nginx

#25

I've been administering Wordpress blogs for almost a decade now; hosting for ... 5? 6 years? I'm not sure any more. Anyhow. I started with Apache. Apache probably works fine if you load the mpm-oh-you-didnt-know-about-mpm-so-sorry module. My next stop was lighttpd. A fine piece of software, with a persistent bug that caused it to drop into a 500 state if FastCGI instances were ever unavailable. So now I'm on Nginx. A…

I'd be interested in seeing your 'fancy footwork' for the rewrite rules :)

Re: Wordpress.com serves 70k requests/second using Nginx

#26
post #25

I've been administering Wordpress blogs for almost a decade now; hosting for ... 5? 6 years? I'm not sure any more. Anyhow. I started with Apache. Apache probably works fine if you load the mpm-oh-you-didnt-know-about-mpm-so-sorry module. My next stop was lighttpd. A fine piece of software, with a persistent bug that caused it to drop into a 500 state if FastCGI instances were ever unavailable. So now I'm on Nginx. A…

I'd be interested in seeing your 'fancy footwork' for the rewrite rules :)

Same here - interested to see that config :) --

On a side note - you're completely right - whoever use nginx as a reverse proxy should just stop doing so and transition to nginx and drop apache.

It's not that Apache isn't good - it's just that Nginx is just way better.

Re: Wordpress.com serves 70k requests/second using Nginx

#27
post #25

I've been administering Wordpress blogs for almost a decade now; hosting for ... 5? 6 years? I'm not sure any more. Anyhow. I started with Apache. Apache probably works fine if you load the mpm-oh-you-didnt-know-about-mpm-so-sorry module. My next stop was lighttpd. A fine piece of software, with a persistent bug that caused it to drop into a 500 state if FastCGI instances were ever unavailable. So now I'm on Nginx. A…

I'd be interested in seeing your 'fancy footwork' for the rewrite rules :)

Edit: actually, it looks like this doesn't work the way I think it does ... looks like nginx has changed enough between 0.7 and 1.0 that my original config doesn't work as correctly as it used to.

Edit 2: Actually I think WP Supercache is the one that changed.

Edit 3: This config looks more up-to-date than mine -- http://rtcamp.com/tutorials/nginx-wordpressmultisite-subdoma...

I particularly liked how they used the try_file directive to cut down on the if statements.

----

Like most people I've cobbled together what I found through Google searching. My case is further complicated because I run a multisite configuration and use the WP Domain Mapping plugin.

It works in a few stages.

WP Supercache is configured to gzip pages to disk. It used to be that you needed to add extra code by hand to support domain mapping, but WP Supercache now cooperates and puts on-disk cached files for each blog in its own directory.

Then I put specific directives in a sites-available/ config file, not the main Nginx file (I serve a static site off the same server).

First,

    server {
        listen  default_server;
The default_server directive means that if another listen directive doesn't pick up an incoming request, it will be handled by this config. Otherwise multisite goes kerflooie.

I like my logs to be divided by site, so:

        access_log  /var/www/log/wordpress/.access.log;
        error_log   /var/www/log/wordpress/.error.log;
Getting end-to-end UTF8 on a stack is a hassle because you have to do it in the database (multiple times, MySQL has about two hojillion charset configuration dials), in PHP and then in Nginx:

        charset utf-8;
Then the obvious:

        root /var/www/wordpress;

        error_page 500 501 502 503 504 = /50x.html;
        location = /50x.html {
            root /var/www/wordpress;
        }
Now for the meat:

        location / {
            # Add trailing slash to */wp-admin requests.
            rewrite /wp-admin$ $scheme://$host$uri/ permanent;
The rewrite is just a little tweak. If you go to /wp-admin, Wordpress will often redirect to the home page. If you go to /wp-admin/, it does what you expect. So this rewrite just adds a trailing slash.

            index  index.php;
For multisite, the Wordpress coders change the URLs files are served from, for reasons that are simply beyond my mortal comprehension. Anyhow, you need a rewrite rule for /files/ URLs:

            # Redirect /file/ URLs.
            rewrite    ^.*/files/(.*)    /wp-includes/ms-files.php?file=$1 last;

Then what follows is based on the common config file you'll see on a dozen blog and forum posts if you google around.

The next thing to do is try and serve a file straight off disk. Nginx does things in an unintuitive order, so even though this rule is further down, it will often fire first:

           # Rewrite URLs for WP-Super-Cache files
           # if the requested file exists, return it immediately
            # this covers the static files case
            if (-f $request_filename) {
                expires 15d;
                break;
            }
The "break" directive basically says, "Oh you found whatever.css|jpg|js? Just serve that up kthxbai".

Now we proceed to determine whether or not we'll try to serve a cached file off disk:

            set $supercache_file '';
            set $supercache_uri $request_uri;

            # don't interfere with POST events (ie form submissions)
            if ($request_method = POST) {
                    set $supercache_uri '';
            }

            # Using pretty permalinks, so bypass the cache for any query string
            if ($query_string) {
                    set $supercache_uri '';
            }

            # Don't show cached version to logged-in users
            if ($http_cookie ~* "comment_author_|wordpress|wp-postpass_" ) {
                    set $supercache_uri '';
            }
If there's still a supercache URL, we try to serve it straight off disk:

            # if we haven't bypassed the cache, specify our supercache file
            if ($supercache_uri ~ ^(.+)$) {
                    set $supercache_file /wp-content/cache/supercache/$http_host/$1index.html;
            }

            # only rewrite to the supercache file if it actually exists
            if (-f $document_root$supercache_file) {
                    rewrite ^(.*)$ $supercache_file break;
            }
Otherwise, give up and let Wordpress grind out the file:

          # all other requests go to Wordpress
          if (!-e $request_filename) {
              rewrite ^.+?(/wp-.*) $1 last;
              rewrite ^.+?(/.*\.php)$ $1 last;
              rewrite ^ /index.php last;
            }
    }

    location ~ .php$ {
        fastcgi_pass  127.0.0.1:9000;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME /var/www/wordpress$fastcgi_script_name;
        include /etc/nginx/fastcgi_params;
    }
From the Wordpress wiki I picked up a few nifty directives to make log files a little less cluttered; I've thrown those in a common file which I include here:

      include /etc/nginx/clean.conf;
    }
clean.conf looks like this:

    # Global restrictions configuration file.
    # Designed to be included in any server {} block.

location = /favicon.ico { log_not_found off; access_log off; } location = /robots.txt { allow all; log_not_found off; access_log off; } # Deny all attempts to access hidden files such as .htaccess, .htpasswd, .DS_Store (Mac). # Keep logging the requests to parse later (or to pass to firewall utilities such as fail2ban) location ~ /\. { deny all; } # Deny access to any files with a .php extension in the uploads directory # Works in sub-directory installs and also in multisite network # Keep logging the requests to parse later (or to pass to firewall utilities such as fail2ban) location ~* /(?:uploads|files)/.*\.php$ { deny all; }
The final step is to add this magic directive to your master nginx.conf:

    gzip_static on;
This tells Nginx to look for a file with a .gz extension. So when it gets redirected to look at whatever-blog.com/index.html, it will also check for index.html.gz. If it finds that file, it will serve it instead.

Re: Wordpress.com serves 70k requests/second using Nginx

#29

I've been administering Wordpress blogs for almost a decade now; hosting for ... 5? 6 years? I'm not sure any more. Anyhow. I started with Apache. Apache probably works fine if you load the mpm-oh-you-didnt-know-about-mpm-so-sorry module. My next stop was lighttpd. A fine piece of software, with a persistent bug that caused it to drop into a 500 state if FastCGI instances were ever unavailable. So now I'm on Nginx. A…

[deleted]
Post reply on HN