Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloud Server Reboots

status.rackspace.com

Post reply on HN
Print view View original

21–30 of 43 posts

Re: Cloud Server Reboots

#21

Post by notatoad »

lvh wrote: post #15
oasisbob wrote: post #13

I envy the AWS users who enjoyed the rolling reboots (which were AZ aware!) across a small minority of the EC2 fleet. (~10%, yeah?) At some point on Sunday, I'm going to be picking the pieces of our entire stack. Rackspace doesn't even offer anything like availability zones. The last major maintenance they scheduled was over the July 4th weekend -- wasn't happy with that one either.

Re: availability zones; while technically true, I'm not sure that's a fair comparison. Rackspace provides uptime guarantees for the internal network per monthly billing period, and they do actually organize the DC in cells (and yes, they do the roll-outs cell-aware). Those cells are simply not end-user visible (AFAIK).

if the "cells" aren't visible in such a way that you can distribute your application across them, it doesn't really matter that they exist.

Re: Cloud Server Reboots

#22

Post by lvh »

notatoad wrote: post #21
lvh wrote: post #15

Earlier quoted context omitted.

Re: availability zones; while technically true, I'm not sure that's a fair comparison. Rackspace provides uptime guarantees for the internal network per monthly billing period, and they do actually organize the DC in cells (and yes, they do the roll-outs cell-aware). Those cells are simply not end-user visible (AFAIK).

if the "cells" aren't visible in such a way that you can distribute your application across them, it doesn't really matter that they exist.

How so? It seems like a sufficiently smart comp^H^H^H^Hservice could figure out how to distribute machines across them, even if the user doesn't know they exist.

Turns out it's a moot point anyway, because cells are client-visible, and have been contributed upstream to OpenStack as a Nova extension.

Re: Cloud Server Reboots

#23

Post by Kudos »

tomjen3 wrote: post #17

Well that is a service I won't use then. First of all you communicate all details so I know how I will be affected. Secondly you don't shut down my service ever, for any reason, other than lack of payment. If you can't do those things you don't get to claim to have excellent support.

I don't think you understand the gravity of the situation. It's a choice between rebooting Xen hosts or having them (and you) get owned when the exploit is made public.

Re: Cloud Server Reboots

#24

Post by ownagefool »

jey wrote: post #7
patio11 wrote: post #3

Those of you who are good at sysadminning don't need this advice, but for the fellow people who are only borderline competent in the room, pay particular attention to this reboot if you recently did "apt-get update" or similar to take care of the bash problem. I've shot myself in the foot before and accepted new updates to e.g. mysql that caused the existing config file to raise a hard error on load, which was only d…

Sounds like it'd be a good idea to perform a deliberate reboot during off-peak hours and preemptively fix any issues.

Rather than doing that, manage your server(s) with code. While there is a little overhead in learning something like ansible, being able to provision another machine to do a dry-run more than makes up for that cost.

Re: Cloud Server Reboots

#25

Post by mikegioia »

csmitheu wrote: post #18
chrissnell wrote: post #2

The timing of this announcement stinks. 2130 PDT on a Friday night, long after most folks have gone home. Making it even more painful, Rackspace is providing 1 hour advance notification. For those of us hosted in the US, there's a rolling reboot window that starts at 0400 PDT on Sunday morning. So, if you're a Rackspace customer and care that your app shuts down cleanly and restarts properly, you get to wake up at 04…

Sorry but in this case your application should have been tested to withstand all possible failure scenarios and restart scenarios so I have little sympathy. It genuinely sounds like you don't trust it to come back up which is not something I could sleep on. Things fail, sometimes violently. Even with our company which has in in-house ops team and our own colocated kit, we expect process failures and restarts and plan…

You're missing his point, and it's condescending/naive of you to assume that his app hasn't been tested to withstand it with no information to tell you that. My app has been thoroughly tested to withstand reboots; however, I have 22 machines with rackspace and (a) I've never tested all 22 going down at random times, and (b) I STILL want to be awake in case anything happens unexpectedly. The pain here is the timing and announcement window for those of us who want to monitor their machines for problems.

Re: Cloud Server Reboots

#26

Post by mikegioia »

patio11 wrote: post #3

Those of you who are good at sysadminning don't need this advice, but for the fellow people who are only borderline competent in the room, pay particular attention to this reboot if you recently did "apt-get update" or similar to take care of the bash problem. I've shot myself in the foot before and accepted new updates to e.g. mysql that caused the existing config file to raise a hard error on load, which was only d…

    pay particular attention to this reboot if you recently
    did "apt-get update" or similar to take care of the bash
    problem
I don't understand. apt-get update won't install new software on reboot. That just updates the index in apt. There's no other way to update bash without "apt-get update && apt-get install bash" and that won't impact anything on reboot.

Your updated mysql config file sounds like you apt-get upgrade'd, installed a new version of mysql, or something like that and the changes didn't go into effect until the mysql process was restarted. I got bit in the ass with that same thing when I apt-get upgrade'd.

Re: Cloud Server Reboots

#27

Post by chippy »

DrJ wrote: post #10

the Xen vulnerability must be something severe if they are all doing this. [1] http://xenbits.xen.org/xsa/

just imagine what all those ISPs and hosts who are not on the private list but who use Xen are thinking. They must be crapping themselves, hitting F5 on the announcements page, ready to pounce and fix.

Re: Cloud Server Reboots

#28

Post by praseodym »

And this is why live migrations (VMware vMotion, but also done by Google Compute Engine) are so awesome. Migrate VMs from server X to server Y, then patch and reboot server X. No VM downtime.

Re: Cloud Server Reboots

#29

Post by pcarmichael »

kam wrote: post #5
click170 wrote: post #4

Earlier quoted context omitted.

The timing of this and the wording of their announcement leads me to think that they weren't included in the select group of people who are brought up to speed with the Xen security update before it's released. The next question then is, who does have access to this? It definitely needs to be kept wrapped-up at least until the large vendors are patched, but who gets to decide who's a large vendor?

The Xen pre-disclosure list is public, and Rackspace is on it: http://www.xenproject.org/security-policy.html They also spell out the policy for who is eligible.

Any word on if/when Linode may schedule reboots?

Re: Cloud Server Reboots

#30

Post by kbar13 »

pcarmichael wrote: post #29
kam wrote: post #5

Earlier quoted context omitted.

The Xen pre-disclosure list is public, and Rackspace is on it: http://www.xenproject.org/security-policy.html They also spell out the policy for who is eligible.

Any word on if/when Linode may schedule reboots?

https://forum.linode.com/viewtopic.php?p=64914#p64914
Post reply on HN