361–368 of 368 posts
DRY "The inconsistency was triggered by a timing quirk in the IP block removal - the IP block had been removed from one configuration file, but this change had not yet propagated to a second configuration file also used in network configuration management."
I'm waiting for the time when they push over the air updates to airplanes in flight. "You can fly safely, we have canaries and staged deployment" A year forward: "Unfortunately because the canary verification as well as the staged deployment code was broken, instead of one crash and 300 dead, an update was pushed to all aircraft, which subsequently caused them to crash, killing 70,000 people." I'm not 100% sure why t…
Yeah ,the part with canary code rub me the wrong way too. These safeguards include a canary step where the configuration is deployed at a single site and that site is verified to still be working correctly This sounds very unprofessional imho. "Touch this cable to see if there is electricity running" sort of thing. Is that really how its should be done?
The thing that stood out for me was: "...team...worked in shifts overnight..."
(Usual disclaimer: I speak for myself, not for my employer, etc.) The team in charge of solving this particular problem is located in two sites in two different timezones. This is true of most critical SRE teams at Google, and it is precisely to be able to have 24h coverage in these time sensitive situations. In the 2+ years I have spent in SRE I have never heard of a single instance of an SRE being asked or even enc…
> However, in this instance a previously-unseen software bug was triggered, and instead of retaining the previous known good configuration, the management software instead removed all GCE IP blocks from the new configuration and began to push this new, incomplete configuration to the network. Always test your crash / exception handling / special case termination+recovery code in production. I have seen this too often…
Earlier quoted context omitted.
Well, how to phrase the same thing briefly without sounding snarky?
You only need to change a few words: "In other words, they simply didn't test their (global!) custom route announcement management software. An edge case was triggered in production, and unsurprisingly they automatically went offline."
Earlier quoted context omitted.
Where is that figure coming from? I'm curious what kinds of failures they have.
https://news.ycombinator.com/item?id=11492569 <- I detailed it a bit more in this comment, including linking the source report from Google.
“Immediate manual control” disengage thresholds are set conservatively. Our objective is not to minimize disengages; rather, it is to gather as much data as possible to enable us to improve our self-driving system.
Also, table 4 reports the number of disengagements (for any reason) each month, as well as the miles driven each month. In the most recent month in that table, it's actually 16 disengagements over 43275.9 miles. That's approximately one disengagement every 2705 miles; about the distance from Sacramento, CA to Washington, DC. At the start of 2015 it was only 343 miles per disengagement; 53 disengagements over 18192.1 miles. The pace of improvement is incredible, especially considering disengagements are set conservatively.
Can a human drive from Sacramento CA to Washington DC without a single close call or mistake along the way? I really doubt it. This technology will be saving lives soon.
Earlier quoted context omitted.
>I don't think that there is any precedent for that sort of thing with manually driven cars. A stroke or heart attack while driving?
A stroke or heart attack while driving has (possibly) nationwide or worldwide reach? I somehow don't think you understand what I'm getting at. A "Perfect Storm" of bugs could cause a systemic failure that has the possibility to affect all cars everywhere (well, probably limited to a single car-maker / model / etc). This has the possibility to affect millions . Claiming that a stroke or heart attack while driving has…
So in that respect, the introduction of self-driving-cars won't necessarily make such events more likely.