Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloud Server Reboots

status.rackspace.com

Post reply on HN
Print view View original

1–10 of 43 posts

Re: Cloud Server Reboots

#2

Post by chrissnell »

The timing of this announcement stinks. 2130 PDT on a Friday night, long after most folks have gone home. Making it even more painful, Rackspace is providing 1 hour advance notification. For those of us hosted in the US, there's a rolling reboot window that starts at 0400 PDT on Sunday morning. So, if you're a Rackspace customer and care that your app shuts down cleanly and restarts properly, you get to wake up at 0400 PDT and check your email and stay near your laptop and internet at least once an hour, every hour, until (potentially) 0400 PDT on Monday morning. Hooray, my Sunday plans are fucked!

They suggest taking backups/snapshots of your instances before the reboot window. Given the throughput required to push multi-hundred GB images from public cloud servers to Cloud Files for storage, I am willing to bet that the backup network is maxed out and will stay maxed out until the outage.

I wonder if Rackspace found out about the rumored Xen exploit from the same people that told Amazon, or if Amazon told Rackspace but waited a little bit to make it more painful for Rackspace's customers...

Re: Cloud Server Reboots

#3

Post by patio11 »

Those of you who are good at sysadminning don't need this advice, but for the fellow people who are only borderline competent in the room, pay particular attention to this reboot if you recently did "apt-get update" or similar to take care of the bash problem. I've shot myself in the foot before and accepted new updates to e.g. mysql that caused the existing config file to raise a hard error on load, which was only discovered the next time the server was restarted. As you can imagine, inability to boot the database has unpleasant consequences for web apps connected to it.

Re: Cloud Server Reboots

#4

Post by click170 »

chrissnell wrote: post #2

The timing of this announcement stinks. 2130 PDT on a Friday night, long after most folks have gone home. Making it even more painful, Rackspace is providing 1 hour advance notification. For those of us hosted in the US, there's a rolling reboot window that starts at 0400 PDT on Sunday morning. So, if you're a Rackspace customer and care that your app shuts down cleanly and restarts properly, you get to wake up at 04…

The timing of this and the wording of their announcement leads me to think that they weren't included in the select group of people who are brought up to speed with the Xen security update before it's released.

The next question then is, who does have access to this? It definitely needs to be kept wrapped-up at least until the large vendors are patched, but who gets to decide who's a large vendor?

Re: Cloud Server Reboots

#5

Post by kam »

click170 wrote: post #4
chrissnell wrote: post #2

The timing of this announcement stinks. 2130 PDT on a Friday night, long after most folks have gone home. Making it even more painful, Rackspace is providing 1 hour advance notification. For those of us hosted in the US, there's a rolling reboot window that starts at 0400 PDT on Sunday morning. So, if you're a Rackspace customer and care that your app shuts down cleanly and restarts properly, you get to wake up at 04…

The timing of this and the wording of their announcement leads me to think that they weren't included in the select group of people who are brought up to speed with the Xen security update before it's released. The next question then is, who does have access to this? It definitely needs to be kept wrapped-up at least until the large vendors are patched, but who gets to decide who's a large vendor?

The Xen pre-disclosure list is public, and Rackspace is on it:

http://www.xenproject.org/security-policy.html

They also spell out the policy for who is eligible.

Re: Cloud Server Reboots

#6

Post by chrissnell »

kam wrote: post #5
click170 wrote: post #4

Earlier quoted context omitted.

The timing of this and the wording of their announcement leads me to think that they weren't included in the select group of people who are brought up to speed with the Xen security update before it's released. The next question then is, who does have access to this? It definitely needs to be kept wrapped-up at least until the large vendors are patched, but who gets to decide who's a large vendor?

The Xen pre-disclosure list is public, and Rackspace is on it: http://www.xenproject.org/security-policy.html They also spell out the policy for who is eligible.

Thanks for the link. Given that Amazon was notifying customers as early as two days ago, we can assume that Rackspace received notification from Xen at least two days ago. They waited until Friday night to announce this? Awful.

Re: Cloud Server Reboots

#7

Post by jey »

patio11 wrote: post #3

Those of you who are good at sysadminning don't need this advice, but for the fellow people who are only borderline competent in the room, pay particular attention to this reboot if you recently did "apt-get update" or similar to take care of the bash problem. I've shot myself in the foot before and accepted new updates to e.g. mysql that caused the existing config file to raise a hard error on load, which was only d…

Sounds like it'd be a good idea to perform a deliberate reboot during off-peak hours and preemptively fix any issues.

Re: Cloud Server Reboots

#8

Post by xelfer »

With AWS and shell shock, it's an interesting time to have just started working for a managed hosting company, nothing like being thrown straight into the deep end of being on call!
Post reply on HN