Earlier quoted context omitted.
I'm guessing they didn't disclose the vendor because they didn't want to be sued for defamation.
Truth is an absolute defense to libel in the US.
January 28th Incident Report
191–198 of 198 posts
Re: January 28th Incident Report
#192Earlier quoted context omitted.
Truth is an absolute defense to libel in the US.
It doesn't stop you from getting sued, though, it merely stops you from losing. It's pretty reasonable to want to avoid a lawsuit you're absolutely certain you could win.
Re: January 28th Incident Report
#193Earlier quoted context omitted.
This seems precisely wrong. Some reading: http://web.mit.edu/Saltzer/www/publications/endtoend/endtoen... https://www.jwz.org/doc/worse-is-better.html [thanks for the hint 'thinkpad20! I don't know what I was thinking.]
It is not precisely wrong, and thanks for tricking me into opening an obscene picture at work, asshole. The internet is designed to be highly fault tolerant, because it was based on an arpanet project to design a network that would NOT go down, even if there was damage to a significant percentage of nodes.
https://web.stanford.edu/class/cs240/old/sp2014/readings/wor...
It's funny, my original comment had the links in plaintext so copying-and-pasting was required and Referer wasn't involved. I changed that on request. b^)
Re: January 28th Incident Report
#194There's no mention of why they don't have redundant systems in more than one datacenter. As they say, it is unavoidable to have power or connectivity disruptions in a datacenter. This is why reliable configurations have redundancy in another datacenter elsewhere in the world.
It's shocking that they don't at least have a read replica of their system in another 'AZ'. That's cloud hosting 101, and being self-hosted isn't an excuse to skimp on this. If an outage caused 2 hours of read-only access to repos it would still be moderately impactful, but at least we could still build our Go code.
Re: January 28th Incident Report
#195"We identified the hardware issue resulting in servers being unable to view their own drives after power-cycling as a known firmware issue that we are updating across our fleet." Tell us which vendor shipped that firmware, so everyone else can stop buying from them.
I'm guessing they didn't disclose the vendor because they didn't want to be sued for defamation.
Re: January 28th Incident Report
#196Re: January 28th Incident Report
#197Earlier quoted context omitted.
I'm on business support too and generally am talking to a rep in minutes. They aren't always able to find the problem before I do, but I always get follow up details later on the how / why that they did determine.
I wish our experience was like this. We used to have business level but we dropped it because we weren't getting value for it. Our experience was slightly better when we had it but we still ended up either fixing most problems on our own or waiting them out.
Re: January 28th Incident Report
#198Every time I read about a massive systems failure, I think of Jurassic Park and am mildly grateful that the velociraptor padock wasn't depending on the systems operation.
Well as long as you're not Samuel L. Jackson in that scenario you should be fine. Ish.
Including the principle that if your software breaks, you're the on who has to go get savaged by velociraptors to fix it.