Cloudflare API Down

We had an outage a few years ago on Black Friday; we had been getting our DC to purchase and rack servers for us for years, and the data centre had IIRC four separate circuits that our servers were on depending on which rack they were in. Unfortunately, we hadn't provided them input into which servers were which purpose, and we occasionally repurposed hardware from one service to another.

This resulted in one of our microservices existing entirely on one circuit, along with significant swaths of other services. We also overprovisioned our hardware so that we could cope with huge increases in traffic, e.g. Black Friday/Cyber Monday weekend. Generally a good idea, but since our DC obviously didn't have any visibility into our utilization, they didn't realize that, if our servers suddenly spiked to 100% CPU use, it could triple our power usage.

Easy to see where this is going, I'm sure.

The microservice which existed entirely on one circuit was one of the most important, and was hit constantly - we were a mobile game company, and this service kept track of players' inventories, etc. Not something you want to hit the database for, so we had layers of caching in Redis and memcached, all of which lived on the application servers themselves, all of them clustered so that we could withstand several of our servers going offline. This meant that when we got a massive influx of players all logging in to take advantage of those Black Friday deals, the service hit probably the hardest was this service, and its associated redis and memcached clusters, as well as (to a lesser extent) the primary database and the replication nodes - some of which were also on the same circuit.

So as we're all trying to tune the systems live to optimize for this large influx of traffic, it trips the breaker on that circuit and something like 1/3 of our servers go offline. We call the CEO of the DC company, he has to call to figure out what the heck just happened, and it takes a while to figure out what the heck just went on and why. Someone has to go into the DC to flip the breaker (once they know that it's not just going to fly again), which is a several-hour drive from Vancouver to Seattle.

Meantime, we all have to frantically promote replication databases, re-deploy services to other application servers, and basically try to keep our entire system up and online for the largest amount of traffic we've ever had on 60% of the server capacity we'd planned on.

I was working on that problem (not just awake, but specifically working on that issue) for 23 hours straight after the power went out. Our CEO made a list of every server we had and how we wanted to balance them across the circuits, and then the DC CEO and I spent all night powering off servers one by one, physically moving them to different cabs, bringing them online, rinse repeat.

TL;DR electricity is complicated.

There's two parts to the answer to GP's question. One is egress filtering, which is widely deployed, and the other is BGP security, which as you know is being deployed.

I assume your site to site latency's under 100ms? If so that's fine.

IMHO edge servers are a waste of time that expose conceptual flaws in the internet's design. We should have had free public proxy servers that relayed any publicly-accessible (public domain) static files automagically like Coral CDN, which is difficult to even find information about anymore:

https://en-academic.com/dic.nsf/enwiki/495553

And that caching should have happened in the ISP's backbone or user's devices so that stuff like municipal wifi "just works". Why are we all downloading the same Netflix stream over and over instead of using a protocol more like BitTorrent with a hash tree or content addressable memory using Subresource Integrity (SRI) for stuff like scripts and fonts instead of compiling it all into huge app.js files? Ridiculous.

Short of that, basically all servers should be using something like Varnish cache, which drops in with little configuration, leaving the main lag time the distance between countries:

http://varnish-cache.org/intro/index.html#intro

One Node.js instance can serve like a million users per month. And it only takes 65 ms for light to get anywhere in the world (12,000 miles divided by 186,000 miles per second). Maybe 200 ms if the speed of light is 3 times slower in wires and fiber optic lines. Yet tons of cloud apps run slower than 1990s cgi-bin servers.

Imagine how we could improve the internet's core technologies with even the tiniest fraction of a fraction of the tech sector's profits and any leisure time at all. This is perhaps the greatest disappointment of my life, watching all of the innovations that I thought would happen come about in a private/corporate fashion which weakens our freedom and privacy.

In my experience, you have building UPS on one MDU and General supply on the other. Building UPS will power everything until generators spin up, and if the UPS itself dies then you're still powered from general supply

Did lose one building about 20 years ago when the generator didn't start

But then I assume that any services I have which are marked as three-nines or more have to be provided from multiple buildings to avoid that type of single point of failure. The services that need five-nines also take into account loss of a major city, beyond that there's significant disruption though -- especially with internet provision, as it's unclear what internet would be left in a more widespread loss of infrastructure.