Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DigitalOcean block storage is down

status.digitalocean.com

Post reply on HN
Print view View original

31–40 of 88 posts

Re: DigitalOcean block storage is down

#31

Post by sunsetMurk »

sunasra wrote: post #30

I was always wondering how I can get know proactively if something like this break or some service has an outage. As a result, I have built this tool( http://incidentok.com )

great idea - going to give it a whirl this week.

i'm curious about the slack integration. can you provide some more info on what that looks like? eg. just a message in real-time when it goes down? a daily message of statuses? etc. Any sort of customization w/ it?

I currently use a soup of zapier zaps to take care of this problem.

Re: DigitalOcean block storage is down

#32

Post by astrodust »

toomuchtodo wrote: post #20
rubbingalcohol wrote: post #17

Earlier quoted context omitted.

to be fair, it's dangerous even in the best hands. mistakes happen but business processes need to be in place to prevent catastrophes... every time i see something like this, my inclination is to blame the CTO, not the engineer who pulled the trigger.

A post mortem should always be a place to highlight deficiencies in processes and communicate necessary improvements put into place, not to blame. Blame should only occur if the cadence of outages becomes excessive. Complex systems are tricky, and to err is human. Disclaimer: Ops/infra engineer in a previous life.

I wonder how many outages these days start with something like "kubectl apply" and then things go horribly awry.

Re: DigitalOcean block storage is down

#33

Post by [deleted] »

astrodust wrote: post #32
toomuchtodo wrote: post #20

Earlier quoted context omitted.

A post mortem should always be a place to highlight deficiencies in processes and communicate necessary improvements put into place, not to blame. Blame should only occur if the cadence of outages becomes excessive. Complex systems are tricky, and to err is human. Disclaimer: Ops/infra engineer in a previous life.

I wonder how many outages these days start with something like "kubectl apply" and then things go horribly awry.

[deleted]

Re: DigitalOcean block storage is down

#37

Post by iamsb »

CaliforniaKarl wrote: post #8

Personally, if DO don’t have anything new in a status post, I’d prefer seeing an update that says something like “We are continuing to work on the issue. Nothing new to report. Next update in X minutes.” That is a lot easier for me to parse than the text that someone seems to be copy/pasting in each update.

Would be great if statuspage.io has a button when pushed publishes message similar to your suggestion.

Re: DigitalOcean block storage is down

#40

Post by solotronics »

rubbingalcohol wrote: post #17
astrodust wrote: post #10

Earlier quoted context omitted.

tmux is a dangerous tool in the wrong hands.

to be fair, it's dangerous even in the best hands. mistakes happen but business processes need to be in place to prevent catastrophes... every time i see something like this, my inclination is to blame the CTO, not the engineer who pulled the trigger.

We can blame whoever we want but you better believe shit rolls downhill at most places.
Post reply on HN