Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Joyent us-east-1 rebooted due to operator error

help.joyent.com

Post reply on HN
Print view View original

1–10 of 129 posts

Joyent us-east-1 rebooted due to operator error

#1

Post by hypervisor »

Due to an operator error, all compute nodes in us-east-1 were simultaneously rebooted. Some compute nodes are already back up, but due to very high load on the control plane, this is taking some time. We are dedicating all operational and engineering resources to getting this issue resolved, and will be providing a full postmortem on this failure once every compute node and customer VM is online and operational. We will be providing frequent updates until the issue is resolved.

Joyent us-east-1 rebooted due to operator error
help.joyent.com

Re: Joyent us-east-1 rebooted due to operator error

#2

Post by alrs »

Joyent's messaging about "we're cloud, but with perfect uptime" was always broken.

It's mildly gross that the current messaging sounds like they're throwing a sysadmin under the bus. If fat fingers can down a data center, that's an engineering problem.

I care about an object store that never loses data and an API that always has an answer for me, even if it's saying things that I don't want to hear.

99.999 sounds stuck-in-the-90s.

Re: Joyent us-east-1 rebooted due to operator error

#4

Post by bcantrill »

It should go without saying that we're mortified by this. While the immediate cause was operator error, there are broader systemic issues that allowed a fat finger to take down a datacenter. As soon as we reasonably can, we will be providing a full postmortem of this: how this was architecturally possible, what exactly happened, how the system recovered, and what improvements we are/will be making to both the software and to operational procedures to assure that this doesn't happen in the future (and that the recovery is smoother for failure modes of similar scope).

Re: Joyent us-east-1 rebooted due to operator error

#6

Post by llamataboot »

SEJeff wrote: post #3

As I've always said, "You can never protect a system from a stupid person with root". You can limit carnage and mitigate this type of thing, but you can't fully protect against sysadmins doing dumb things (unless you just hire great sysadmins)

I don't think "just hiring great sysadmins" is possible. People have off-days or are tired or sick, new people get on-boarded, even great people make mistakes, etc.

Re: Joyent us-east-1 rebooted due to operator error

#10

Post by cordite »

shiftpgdn wrote: post #7

Let this be a lesson to linux admins. Re-alias shutdown -r now into something else on production servers. I once took down access to about 6000 servers because I ran the script to decommission servers on our jump box when I got the SSH windows confused.

I once put `shutdown -h now` (halt) instead of `shutdown -r now` (reboot)

Once I realized what had happened on the production server I ended up calling OVH (and they were helpful but not immediately acting).

It's not a good feeling.

Post reply on HN