#4 I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4. #1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the ne…
How does the fact you worked at those companies relate to #4? Edit: I misread the parent and my question doesn't make a lot of sense. Please ignore it :)
> How does the fact you worked at those companies relate to #4?
For Facebook I worked on the incident, previous Wednesday. 9.5 hours of pain...
And for my past employers, I still have friends there texting the root causes with facepalm emojis.
#4 I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4. #1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the ne…
How does the fact you worked at those companies relate to #4? Edit: I misread the parent and my question doesn't make a lot of sense. Please ignore it :)
Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…
#4 I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4. #1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the ne…
Wow, such certainty. "I worked at these companies that went down, so I knew and still know everything, I can even rule out the possibility of other organizations which I did not work for screwing with them, because I know everything about those too. And yes, these FANGs are complex organizations, but not so complex that a former employee like myself wouldn't know what the cause of an outage is or at least isn't (Hell, I'd fix it, but I have to finish explaining why complexity is not the reason why they are crashing, and the series of ten dollar words that constitute my explanation aren't exactly the quickest to type."
Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…
Another possibility: US (or other) authorities are requiring some sort of monitoring software or hardware that where disruption of service is unavoidable during install
Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…
#4 I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4. #1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the ne…
Yep, this also matches what I've heard through the grapevine.
Pushing bad regex to production, chaos monkey code causing cascading network failure, etc.
They're just different accidents for different reasons. Maybe it's summer and people are taking more vacation?
A lot of high-profile outages recently. Can't actually remember the last time Twitter went fully down. Have to confess I immediately assumed an issue with my own connection, even though every other site is working. Unrelated, but for some reason the phrase "I have no mouth and I must scream" just popped into my head
It makes me wonder if a powerful malicious agent can devise a complex operation (planting people in several key places) and wipe out most of the databases of one these places. It would be interesting to see what would it happen if suddenly all of twitter of facebook is deleted , puff!, gone in a second.
You can't delete that much data in a matter of seconds digitally. You need to physically harm the hardware.
A lot of high-profile outages recently. Can't actually remember the last time Twitter went fully down. Have to confess I immediately assumed an issue with my own connection, even though every other site is working. Unrelated, but for some reason the phrase "I have no mouth and I must scream" just popped into my head
A lot of high-profile outages recently. Can't actually remember the last time Twitter went fully down. Have to confess I immediately assumed an issue with my own connection, even though every other site is working. Unrelated, but for some reason the phrase "I have no mouth and I must scream" just popped into my head
It makes me wonder if a powerful malicious agent can devise a complex operation (planting people in several key places) and wipe out most of the databases of one these places. It would be interesting to see what would it happen if suddenly all of twitter of facebook is deleted , puff!, gone in a second.
Well, Tim Berners-Lee told that internet is a fragile technology and should be re-made, it wasn't meant to be this big. This is not really much related to these problems, but I think it should fire up this discussion, it is that important.