The title sounds like an AMD cpu issue.
Air traffic failure caused by two locations 3600nm apart sharing 3-letter code
211–220 of 311 posts
Re: Air traffic failure caused by two locations 3600nm apart sharing 3-letter code
#212So, exactly the same airline (French Bee) and exactly the same route (LAX-ORY) and exactly the same waypoint (DVL) as last September, resulting in exactly the same failure mode: https://chaos.social/@russss/111048524540643971 Time to tick that "repeat incident?" box in the incident management system, guys.
It's an article about that same accident
Re: Air traffic failure caused by two locations 3600nm apart sharing 3-letter code
#213Earlier quoted context omitted.
Every once in a while I get an email about a ten plus year old bug finally getting fixed in some open source project. If it's a good bug accurately describing a real thing, there's no reason to throw that work away rather than just marking it lower priority.
> If it's a good bug accurately describing a real thing, there's no reason to throw that work away rather than just marking it lower priority. Perhaps. But the triage to separate the "good bugs accurately describing real things" from the chaff isn't free either.
Storage is cheap, database indexes work. Just add a `TimesChecked` counter to your bug tracker.
Now when considering priorities, consider not just impact and age, but also times checked.
It's less expensive than going and deleting stuff. Unless you're automating deletions? In which case... I don't think I can continue this discussion.
Re: Air traffic failure caused by two locations 3600nm apart sharing 3-letter code
#214Earlier quoted context omitted.
At Google, the global Chubby cell had gone so long without any downtime that people were starting to assume that’s it was just always available, leading to some kind of outage or other when the global cell finally did have some organic downtime. Chubby-SRE added quarterly synthetic downtime of the global cell (iff the downtime SLA had not already been exceeded).
For those of us who haven't worked at Google, what's "Chubby" and what's a "cell"?
I don't work at Google but I do live in a country where I have access to google.com. Chubby is a lock service that is used internally at Google and a cell is referring to an instance of Chubby. You can read more here: https://static.googleusercontent.com/media/research.google.c...
Re: Air traffic failure caused by two locations 3600nm apart sharing 3-letter code
#215Earlier quoted context omitted.
At Google, the global Chubby cell had gone so long without any downtime that people were starting to assume that’s it was just always available, leading to some kind of outage or other when the global cell finally did have some organic downtime. Chubby-SRE added quarterly synthetic downtime of the global cell (iff the downtime SLA had not already been exceeded).
For those of us who haven't worked at Google, what's "Chubby" and what's a "cell"?
Replace this with "API gateway cluster", or basically any simple enough, very widely used service.
Re: Air traffic failure caused by two locations 3600nm apart sharing 3-letter code
#216oh nautical miles !
not nanometres as you might assume from being used to normal units
Re: Air traffic failure caused by two locations 3600nm apart sharing 3-letter code
#217Earlier quoted context omitted.
At Google, the global Chubby cell had gone so long without any downtime that people were starting to assume that’s it was just always available, leading to some kind of outage or other when the global cell finally did have some organic downtime. Chubby-SRE added quarterly synthetic downtime of the global cell (iff the downtime SLA had not already been exceeded).
For those of us who haven't worked at Google, what's "Chubby" and what's a "cell"?
Ah, chubby is a distributed lock service. Think “zookeeper” and you won’t be far off.
https://static.googleusercontent.com/media/research.google.c... [pdf]
Some random blog post: https://medium.com/coinmonks/chubby-a-centralized-lock-servi...
You can run multiple copies/instances of chubby at the same time (like you could run two separate zookeepers). You usually run an odd number of them, typically 5. A group of chubby processes all managing the same namespace is a “cell”.
A while ago, nearly everything at Google had at least an indirect dependency on chubby being available (for service discovery etc), so part of the standard bringup for a datacenter was setting up a dc-specific chubby cell. You could have multiple SRE-managed chubby cells per datacenter/cluster if there was some reason for it. Anybody could run their own, but chubby-sre wasn’t responsible for anybody else’s, I think.
Finally, there was a global cell. It was both distributed across multiple datacenters and also contained endpoint information for the per-dc chubby cells, so if a brand new process woke up somewhere and all it knew how to access was the global chubby cell, it could bootstrap from that to talking to chubby in any datacenter and thence to any other process anywhere, more or less.
^ there’s a lot in there that I’m fuzzy about, maybe processes wake up and only know how to access local chubby, but that cell has endpoint info for the global one? I don’t think any part of this process used dns; service discovery (including how to discover the service discovery service) was done through chubby.
Re: Air traffic failure caused by two locations 3600nm apart sharing 3-letter code
#218Earlier quoted context omitted.
True, but it's the exception that proves the rule; it's also the same company responsible for delivering a staggeringly high percentage of internet video, typically without a hitch.
That's not what an exception proving a rule means. It has a technical meaning: a sign that says "free parking on sundays" implies parking is not free as a rule. When used like this it just confuses a reader with rethoric. In this case netflix is just bad at live streaming, they clearly haven't done the necessary engineering work on it.
The fact that Netflix surprised so many people by an exceptional technical issue implies that as a rule Netflix delivers video smoothly and at any scale necessary.
Re: Air traffic failure caused by two locations 3600nm apart sharing 3-letter code
#219Earlier quoted context omitted.
Which company deployed a chaos monkey deamon on their systems? Seemed to improve resiliency when I read about it.
"Your flight has been delayed due to Chaos Monkey."
This means a major system problem. The point of the Chaos Monkey is that the system should function without interruptions or problems despite the activity of the Chaos Monkey. That is, it is to keep the system in such a shape that it could swallow and overcome some rate of failure, higher than the "naturally occurring" rate.
Re: Air traffic failure caused by two locations 3600nm apart sharing 3-letter code
#220Earlier quoted context omitted.
> which results in more low-speed, surface-street collisions, while there’s no change in high speed collisions because a short spell of distractedness on the highway is less likely to result in an accident. That's not a remotely plausible model though. There are recorded cases of e.g. 1.6 seconds of distractedness at high speed causing a fatal collision. Anything that increases road injuries is almost certainly also…
Yeah, I was not really trying to argue that that was actually the case, so won’t waste time trying to defend the merits of the model I pulled out of my arse. Alternatively then, perhaps safety developments in cars made them safer to drive around the same time? Or maybe advances in medicine made fatal crashes less likely? Or perhaps there’s some other explanation that doesn’t immediately spring to mind, it’s irrelevan…
> The only point I’m really making is that the data OP referred to does not show an increase in excess deaths, and in fact specifically fails to find this.
That data absolutely does show an increase in deaths, it's right there in the table. It fails to find a statistically significant increase in deaths. The most plausible explanation for that is that the study is underpowered because of the sample size, not that some mystical effect increased injuries without increasing deaths.