Live data from Hacker News

January 28th Incident Report

github.com

191–198 of 198 posts

Re: January 28th Incident Report

#191
post #151

Earlier quoted context omitted.

I'm guessing they didn't disclose the vendor because they didn't want to be sued for defamation.

Truth is an absolute defense to libel in the US.

It doesn't stop you from getting sued, though, it merely stops you from losing. It's pretty reasonable to want to avoid a lawsuit you're absolutely certain you could win.

Re: January 28th Incident Report

#192

Earlier quoted context omitted.

Truth is an absolute defense to libel in the US.

It doesn't stop you from getting sued, though, it merely stops you from losing. It's pretty reasonable to want to avoid a lawsuit you're absolutely certain you could win.

Vendors very seldom sue customers for publicly saying their product is defective. The negative publicity tends to backfire. Legal action can backfire even worse. If the vendor claims the product isn't defective, they have to prove that in court to win a libel action. That means discovery and examination of the company's internal documents and the complaints of other customers, all on the record.

Re: January 28th Incident Report

#193

Earlier quoted context omitted.

This seems precisely wrong. Some reading: http://web.mit.edu/Saltzer/www/publications/endtoend/endtoen... https://www.jwz.org/doc/worse-is-better.html [thanks for the hint 'thinkpad20! I don't know what I was thinking.]

It is not precisely wrong, and thanks for tricking me into opening an obscene picture at work, asshole. The internet is designed to be highly fault tolerant, because it was based on an arpanet project to design a network that would NOT go down, even if there was damage to a significant percentage of nodes.

The "asshole" in this case is JWZ, [randomly?] switching on the Referer header. Apparently he has a hard-on for HN; he's not the only one, but I won't be linking to his site again. (Although, is that really "obscene"? It doesn't do anything for me?) Try this instead, since Stanford are unlikely to engage in such shenanigans:

https://web.stanford.edu/class/cs240/old/sp2014/readings/wor...

It's funny, my original comment had the links in plaintext so copying-and-pasting was required and Referer wasn't involved. I changed that on request. b^)

Re: January 28th Incident Report

#194
post #67
post #2

There's no mention of why they don't have redundant systems in more than one datacenter. As they say, it is unavoidable to have power or connectivity disruptions in a datacenter. This is why reliable configurations have redundancy in another datacenter elsewhere in the world.

It's shocking that they don't at least have a read replica of their system in another 'AZ'. That's cloud hosting 101, and being self-hosted isn't an excuse to skimp on this. If an outage caused 2 hours of read-only access to repos it would still be moderately impactful, but at least we could still build our Go code.

So your building process depends on the availability of an external company?

Re: January 28th Incident Report

#195
post #151

"We identified the hardware issue resulting in servers being unable to view their own drives after power-cycling as a known firmware issue that we are updating across our fleet." Tell us which vendor shipped that firmware, so everyone else can stop buying from them.

I'm guessing they didn't disclose the vendor because they didn't want to be sued for defamation.

And/or they want to maintain a working relationship with said vendor. Going nuclear is a good way of getting _exactly_ the minimum level of service that your SLA specifies.

Re: January 28th Incident Report

#197

Earlier quoted context omitted.

I'm on business support too and generally am talking to a rep in minutes. They aren't always able to find the problem before I do, but I always get follow up details later on the how / why that they did determine.

I wish our experience was like this. We used to have business level but we dropped it because we weren't getting value for it. Our experience was slightly better when we had it but we still ended up either fixing most problems on our own or waiting them out.

How much do you pay per month for AWS? That might be a difference.

Re: January 28th Incident Report

#198

Every time I read about a massive systems failure, I think of Jurassic Park and am mildly grateful that the velociraptor padock wasn't depending on the systems operation.

Well as long as you're not Samuel L. Jackson in that scenario you should be fine. Ish.

Samuel L. Jackson taught me everything I know about ethics in software engineering.

Including the principle that if your software breaks, you're the on who has to go get savaged by velociraptors to fix it.

Post reply on HN