Live data from Hacker News

Due to the fires and power outages in California, oreilly.com is unavailable

oreilly.com

11–17 of 17 posts

Re: Due to the fires and power outages in California, oreilly.com is unavailable

#11

No geographic redundancy? "Well, this is awkward…" Doesn't quite cover it I think, especially given that these problems are not a bolt out of the blue.

Yeah this is really weird. I remember we had fires in California and one of our Data Centers was at risk. We migrated the apps to another primary DC.

You would think they would move it ASAP if their DC was at risk.

Re: Due to the fires and power outages in California, oreilly.com is unavailable

#13
They include a separate link if you’re looking for online learning, which gives you a login page, so clearly the part you’d care about as a customer is still available.

Since that would also seem to be the most intensive part to keep operational I’m confused...

Re: Due to the fires and power outages in California, oreilly.com is unavailable

#14
post #4

Surprising that they don't have a geo-failover location. When I worked in the DR field many years ago all but the smallest companies had some sort of business continuity plan if disaster strikes, but perhaps this is an example of positive bias as only companies that already cared a lot about DR would get a DR consultancy. Those few companies I saw that had no DR and BCP whatsoever were usually small companies that gr…

Having been through a variety of audits where this is invariably a topic that comes up at couple of different companies you also have to keep in mind that it’s one of those situations where one size doesn’t fit all.

A company with a call center had to have a plan in place for replacing office space (and, yikes, call center personnel) because having people available to answer calls and meet their SLA requirements was the biggest risk. The servers... meh. It wouldn’t be pretty, but grabbing a machine at the nearest Best Buy and picking up one of the semi-recent offsite backups from an external hard drive at the CEO’s house would get everybody through.

At a dev shop with a variety of state and federal government clients the office couldn’t matter less, what was important was geo redundancy, so we had racks in VA and in somewhere out west, I forget where. Another instance where everything probably could have been run (for a few days at least) off a couple off the shelf boxes stuck under someone’s desk, but it would have to be immediately, and no one lived anywhere to do that practically, so the second DC was invested in.

It’s all about identifying your specific risk and figuring out what to do to mitigate it to a reasonable degree without breaking the bank. Apparently Oreilly didn’t think geo redundancy for their main site was worth that trade off.

Re: Due to the fires and power outages in California, oreilly.com is unavailable

#15

They include a separate link if you’re looking for online learning, which gives you a login page, so clearly the part you’d care about as a customer is still available. Since that would also seem to be the most intensive part to keep operational I’m confused...

Last night when I tried it, links to the online learning site (both to the login page and to specific chapters of books I'm in the middle of reading) ended up at the page about the fire. The link to the online learning site from that page just came back to that page, making me wonder what the point was.

Re: Due to the fires and power outages in California, oreilly.com is unavailable

#16
post #4

Surprising that they don't have a geo-failover location. When I worked in the DR field many years ago all but the smallest companies had some sort of business continuity plan if disaster strikes, but perhaps this is an example of positive bias as only companies that already cared a lot about DR would get a DR consultancy. Those few companies I saw that had no DR and BCP whatsoever were usually small companies that gr…

Really surprising. Even if they self-host everything, you can still design a cloud based emergency spin up configuration.

Re: Due to the fires and power outages in California, oreilly.com is unavailable

#17
post #4

Surprising that they don't have a geo-failover location. When I worked in the DR field many years ago all but the smallest companies had some sort of business continuity plan if disaster strikes, but perhaps this is an example of positive bias as only companies that already cared a lot about DR would get a DR consultancy. Those few companies I saw that had no DR and BCP whatsoever were usually small companies that gr…

Having been through a variety of audits where this is invariably a topic that comes up at couple of different companies you also have to keep in mind that it’s one of those situations where one size doesn’t fit all. A company with a call center had to have a plan in place for replacing office space (and, yikes, call center personnel) because having people available to answer calls and meet their SLA requirements was…

As a matter of fact, we're on GKE for most of our stuff, having moved out about 6 months ago. There are some remnants left, however, and that's what's causing the issue today.

The site is online, via learning.oreilly.com. The anonymous pages and login were having issues, because the login was tied to something (not familar with the exact details) in the DC offline.

So if you were already logged in, you won't notice much of anything (hopefully!). But if your session expired and you have to re-log in or were trying to see an preview of a book, that's where the problem existed.

Post reply on HN