281–290 of 306 posts
Earlier quoted context omitted.
Of possible interest: https://www.youtube.com/watch?v=Ijst4g5KFN0 This is a presentation to students by an MIT professor that goes over exactly what happened, the sequence of events, mistakes made, and so on.
Warning for others: I watched the above video and then watched the entire course (>30 hours).
Earlier quoted context omitted.
Agreed, but with an amendment: If your tool is capable of draining the whole universe, period , it is too dangerous to exist. That was one of the big takeaways: global config changes must happen slowly. (Whether we've fully internalized that lesson is a different matter.)
SREs live in a dangerous world, unfortunately. It's entirely possible the "tool" in question is a shell script that gets fed a list of bad cells but some bug causes it to get a list of all the cells instead. Some tools are well engineered, capable of the Sisyphean task of globally deploying updates but others are rapid prototypes that, sure, are too dangerous to exist, but the whole point of SREs being capable progra…
There's solutions to that, but of course these sorts of tools all come into existence well before the system reaches a size where how they work becomes dangerous.
> Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command. I'm so glad to see that they framed this in terms of a bug in a tool designed to prevent human error, rather than simply blaming it on human error.
I wouldn't be surprised if that tool was a shell script with a mistyped conditional somewhere, I really dislike shell scripting.
Earlier quoted context omitted.
The buggy audit tool was probably made by a human too, though.
But reviewed by other humans. At some count, a collective human error becomes a system error.
https://learntla.com/introduction/
I discovered it here on HN just recently, in a comment on a new tool in the same problem space.
Earlier quoted context omitted.
> I'm so glad to see that they framed this in terms of a bug in a tool designed to prevent human error, rather than simply blaming it on human error. Wouldn't human error reflect extremely poorly on the company though? I mean, for human error to be the root cause of this mega-outage, that would imply that the company's infrastructure and operational and security practices were so ineffective that a single person scre…
Organisations have a bad habit of using "human error" to blame systemic problems whose true root cause is inadequate leadership on individual low level employees. So, we're glad to see Facebook didn't try this shitty practice. For a modern example, look for information on Symantec's "A tough day as leaders" in which they try to blame an incident that's clearly a result of at least incompetence by senior management on…
In other words, not human error but human error.
Earlier quoted context omitted.
Organisations have a bad habit of using "human error" to blame systemic problems whose true root cause is inadequate leadership on individual low level employees. So, we're glad to see Facebook didn't try this shitty practice. For a modern example, look for information on Symantec's "A tough day as leaders" in which they try to blame an incident that's clearly a result of at least incompetence by senior management on…
So, not human error, but inadequate leadership, which is also a human error. In other words, not human error but human error.
I'm confused by this. Are the DNS servers inside the backbone, or outside? To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection. In the recent outage the entire backbone was removed from operation, making these locations declare themselves unhealthy and withdraw those BGP adve…
If we make a simplified network map, FB looks more or less like a bunch of PoPs (points of presence) at major peering points around the world, a backbone network that connects those PoPs to the FB datacenters, and the FB operated datacenters themselves. (The datacenters are generally located a bit farther away from population centers, and therefore peering points, so it's sensible to communicate to the outside world…
Earlier quoted context omitted.
I wouldn't be surprised if that tool was a shell script with a mistyped conditional somewhere, I really dislike shell scripting.
As opposed to what? Sixteen pages of boilerplate Java/Python?
That said, even clean, idiomatic Python isn't as terse as sh. It also isn't as terse as perl. Many would argue that's a good thing. The optimum point for readability isn't found at either of the extremes. Not entirely unlike how the most readable way of writing English is neither shorthand nor blackletter.
Earlier quoted context omitted.
> This event left a lot of scar tissue across all of Technical Infrastructure, and the next few months were not a fun time (e.g. a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust). Bullshit. I'd believe this if it was not completely impossible for 99.999999% of google "customers" to contact anyone at the company. Or for the decade and a half of p…
Maps, Mail, Drive, Scholar, and Search are all the best or near the best available. That doesn’t mean I like every one of them or I wouldn’t prefer others, but as far as I can tell the competition doesn’t exist that works better. GCP and Pixel phones are a toss-up between them and competitors. It isn’t market dominance, nobody has made anything better.
(Data errors I’ve seen this week: the aerial imagery over Brisbane Australia is from ~2010 but labeled 2021, the coastline near Barentsburg in Svalbard is wrong and doesn’t match any other map.)
Earlier quoted context omitted.
You are to the point and precise. This is exactly the problem. Each individual PoP did the right thing by dropping out of the anycast, but because they all did it, it was the wrong thing. Somehow I feel the design is flawed because if abuses DNS server status a bit. I mean DNS server down and BGP withdrawal for the DNS server is a perfect combination, however connectivity between DNS and backend server down, DNS up a…
I think this is not necessarily a flaw of the design. It's a fundamental weakness of the real world. Either you can take the backbone being unavailable as a symbol that the PoP is broken, and kill the PoP; or you can take the backbone being unavailable as a symbol that the backbone is broken and do your best. When either interpretation is wrong, you'll need humans to come around and intervene. It's much more common t…
internal tools to bring back the backbone relied on DNS which relied on the backbone being up