371–380 of 544 posts
Earlier quoted context omitted.
Given that today was the White House's "Social Media Summit", no doubt there will be a few conspiracies floated. I'm betting "Twitter wanted to block out all the criticism coming form the summit!" will be a popular one.
I've got a couple conspiracy theories at the moment: 1. It's a deployment of some infrastructure change the government got the big tech companies to sign onto. 2. It's a "shot across the bow" from some external party to demonstrate their control over major infrastructure. 2.a. Also could have been a mix of 1 and 2. The government orchestrated the outages in order to add fuel to the hysteria over Chinese "spy chips".…
Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…
It's #1. The real question is how this isn't blindingly obvious to everyone.
Really curious which part of their infrastructure was the root cause.
Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…
Earlier quoted context omitted.
How does the fact you worked at those companies relate to #4? Edit: I misread the parent and my question doesn't make a lot of sense. Please ignore it :)
> How does the fact you worked at those companies relate to #4? For Facebook I worked on the incident, previous Wednesday. 9.5 hours of pain... And for my past employers, I still have friends there texting the root causes with facepalm emojis.
Earlier quoted context omitted.
Wow, such certainty. "I worked at these companies that went down, so I knew and still know everything, I can even rule out the possibility of other organizations which I did not work for screwing with them, because I know everything about those too. And yes, these FANGs are complex organizations, but not so complex that a former employee like myself wouldn't know what the cause of an outage is or at least isn't (Hell…
geez, tough crowd. do you wanna ten dollar hug?
Earlier quoted context omitted.
Yep, this also matches what I've heard through the grapevine. Pushing bad regex to production, chaos monkey code causing cascading network failure, etc. They're just different accidents for different reasons. Maybe it's summer and people are taking more vacation?
I actually like the summer vacation hypothesis. Makes the most sense to me - backup devs handling some things they are not used to.
Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…
>July 11, 2019 7:56PM UTC[Identified] The outage was due to an internal configuration change, which we're now fixing. Some people may be able to access Twitter again and we're working to make sure Twitter is available to everyone as quickly as possible. Seems #4 is the winner
I work on critical infrastructure at FAANG and it's frightening how complex things are. The folks who built the systems and knew them inside-out have mostly moved on, and the newbies, like me, don't fully understand how the abstractions leak, what changes cause what side effects etc.