Using Flat Files So Elections Don’t Break Your Server

For the regular ebb and flow of data, you can plan, and use varnish/rails/nginx/ all the magic tools you want, and things will work great - but you also have added complexity, and when something goes wrong, the more complex the system, the longer it generally takes to fix. Especially under unexpected heavy load.

Simplifying the system down to flat files and de-coupling the ROR stuff gives you a clear troubleshooting point - if something goes berzerk, you can cut the link between the two and troubleshoot in relative safety before turning replication back on.

I'm rambling - the main point is to reduce the complexity involved in the end-user transaction to be as efficient and fast as possible so you can deal with an unknown load factor coming in on a really important day. Going down that day would be BAD for business.

Apache can, one way or the other, do most of what the others can do - possibly not as well, and at the risk of a much more complex configuration.

Haproxy does one thing and does it really,really well - it's great for dealing with load balancing, concurrency limiting per user defined resource, and identifying and routing incoming requests to the right infrastructure. It's really good at this - that's what it does.

Varnish does one thing and does it really, really well - it caches content and serves it up (usually out of memory, but even if it's swapped, it's optimized) to keep the load off your application servers. It has various optimizations built into make it really good at this.

So - in this case, the answer might be "not much - we could just put the apache's out front behind haproxy" - but it appears putting the varnish server out front with a 5 second cache dropped the load on the application servers (in this case apache serving static files that it receives over rsync - dont' forget the rsync part - resources are needed for that). This might result in smoother output for the end user, rather than something hitting a node that's busy servicing an rsync update. It may also be their engineers are very familiar with the haproxy/varnish front end setup, as it presumably exists in their current day to day operation as well.. so the people responsible for keeping things up probably decided "Yes, we'd like to keep it there - it makes our lives easier."

There is an operational anti-pattern in there - removing too many elements from a known system is also a kind of added cmoplexity - all your troubleshooting methods disappear.

Their goal here was to de-couple the dynamic elements from the event-driven side of things and turn them into something more resilient (and less flexible) for a short time to deal with unknown and unpredicably large load.

As the author points out, having the Times election site go down on election eve would be a BIG PROBLEM -- massive losses in both reputation and advertising revenue. For the system architect, a failure could possibly mean losing his job.

What the author has laid out is a system that is robust to multiple, simultaneous failures (with possible exception of the loss of AWS, although that's not entirely clear). That just seems like good planning.