Live data from Hacker News

DigitalOcean block storage is down

status.digitalocean.com

21–30 of 88 posts

Re: DigitalOcean block storage is down

#21

What unholy thing did they do that broke it across 12 different datacenters, good lord.

This does seem to indicate a notable lack of isolation for the blast radius between DO datacenters. Would be interesting to see the post mortem.

Propagating mistakes across all the things is devops

Re: DigitalOcean block storage is down

#22

What unholy thing did they do that broke it across 12 different datacenters, good lord.

This does seem to indicate a notable lack of isolation for the blast radius between DO datacenters. Would be interesting to see the post mortem.

Google Cloud recently had a global outdate. DevOps tools that interact with all resources across data centers are primarily the culprit.

Re: DigitalOcean block storage is down

#23

Earlier quoted context omitted.

I get the feeling that whoever writes the post-mortem is going to have a bit of pressure to assure folks that there is isolation going forward.

That would be a bad sign that there’s something wrong with the culture. I would hope for a postmortem that identified flaws that genuinely needed to be fixed.

Those are not mutually exclusive and actually a good idea. You want to fix this specific issue, but also ensure that whatever process took down one DC doesn't affect other DCs. That's scaling and redundancy 101 - not sure why it would be something wrong.

Re: DigitalOcean block storage is down

#25

What unholy thing did they do that broke it across 12 different datacenters, good lord.

Thankfully they fired the VP last year who is a complete idiot when it comes to infrastructure. Probably related to decisions he made no doubt. Companies prior to DO are still fixing his bad decisions. And now he's in charge of blockchain stuff. wcpgw????

Re: DigitalOcean block storage is down

#26

Earlier quoted context omitted.

This does seem to indicate a notable lack of isolation for the blast radius between DO datacenters. Would be interesting to see the post mortem.

Propagating mistakes across all the things is devops

DevOps Borat: "To make error is human. To propagate error to all server in automatic way is #devops."

https://twitter.com/DEVOPS_BORAT/status/41587168870797312

Re: DigitalOcean block storage is down

#28

Earlier quoted context omitted.

I get the feeling that whoever writes the post-mortem is going to have a bit of pressure to assure folks that there is isolation going forward.

That would be a bad sign that there’s something wrong with the culture. I would hope for a postmortem that identified flaws that genuinely needed to be fixed.

I would think having all zones go down is a flaw that genuinely needed to be fixed.
Post reply on HN