Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloud Server Reboots

status.rackspace.com

Post reply on HN
Print view View original

11–20 of 43 posts

Re: Cloud Server Reboots

#11

Post by Maxious »

kam wrote: post #5
click170 wrote: post #4

Earlier quoted context omitted.

The timing of this and the wording of their announcement leads me to think that they weren't included in the select group of people who are brought up to speed with the Xen security update before it's released. The next question then is, who does have access to this? It definitely needs to be kept wrapped-up at least until the large vendors are patched, but who gets to decide who's a large vendor?

The Xen pre-disclosure list is public, and Rackspace is on it: http://www.xenproject.org/security-policy.html They also spell out the policy for who is eligible.

prgmr.com also got inducted 2 days ago http://blog.prgmr.com/xenophilia/2014/09/prgmrcom-now-on-xen...

Xen has previously had problems with people leaking embargoed security issues http://www.gossamer-threads.com/lists/xen/devel/248239

Re: Cloud Server Reboots

#12

Post by Maxious »

patio11 wrote: post #3

Those of you who are good at sysadminning don't need this advice, but for the fellow people who are only borderline competent in the room, pay particular attention to this reboot if you recently did "apt-get update" or similar to take care of the bash problem. I've shot myself in the foot before and accepted new updates to e.g. mysql that caused the existing config file to raise a hard error on load, which was only d…

Two things like this; my tomcat config got reset to max 128mb JVM stack and ubuntu seems to have switched from -virtual to -generic kernels but the latter isn't booting for me.

Re: Cloud Server Reboots

#13

Post by oasisbob »

I envy the AWS users who enjoyed the rolling reboots (which were AZ aware!) across a small minority of the EC2 fleet. (~10%, yeah?)

At some point on Sunday, I'm going to be picking the pieces of our entire stack.

Rackspace doesn't even offer anything like availability zones.

The last major maintenance they scheduled was over the July 4th weekend -- wasn't happy with that one either.

Re: Cloud Server Reboots

#14

Post by thaumaturgy »

jey wrote: post #7
patio11 wrote: post #3

Those of you who are good at sysadminning don't need this advice, but for the fellow people who are only borderline competent in the room, pay particular attention to this reboot if you recently did "apt-get update" or similar to take care of the bash problem. I've shot myself in the foot before and accepted new updates to e.g. mysql that caused the existing config file to raise a hard error on load, which was only d…

Sounds like it'd be a good idea to perform a deliberate reboot during off-peak hours and preemptively fix any issues.

You would think so (and I won't say it isn't), but if you're just one guy handling a few servers for your hosted SaaS, it's easy to fall into the "don't disturb the dust" trap -- especially once you've had a routine maintenance reboot or upgrade completely bone you.

Not too long ago I had a 36-hour straight marathon sysadmin session when a routine update happened to break my particular stack without advance notice on a web server used by a bunch of customers. I think my eye still twitches when I think of it.

Usually the next thing people say is, "you should set up x, y, and z tools for better redundancy / failover / load distribution / backups / sysadmin management / etc." Again, they're not wrong, but none of those things directly makes you more money, and making time for them can be difficult, especially since so few of them can be set up as easily as they claim on the box.

Re: Cloud Server Reboots

#15

Post by lvh »

oasisbob wrote: post #13

I envy the AWS users who enjoyed the rolling reboots (which were AZ aware!) across a small minority of the EC2 fleet. (~10%, yeah?) At some point on Sunday, I'm going to be picking the pieces of our entire stack. Rackspace doesn't even offer anything like availability zones. The last major maintenance they scheduled was over the July 4th weekend -- wasn't happy with that one either.

Re: availability zones; while technically true, I'm not sure that's a fair comparison. Rackspace provides uptime guarantees for the internal network per monthly billing period, and they do actually organize the DC in cells (and yes, they do the roll-outs cell-aware). Those cells are simply not end-user visible (AFAIK).

Re: Cloud Server Reboots

#16

Post by jleader »

patio11 wrote: post #3

Those of you who are good at sysadminning don't need this advice, but for the fellow people who are only borderline competent in the room, pay particular attention to this reboot if you recently did "apt-get update" or similar to take care of the bash problem. I've shot myself in the foot before and accepted new updates to e.g. mysql that caused the existing config file to raise a hard error on load, which was only d…

I've never been a "real" sysadmin, but back in the mid 80s I learned that anything you don't practice doing regularly will inevitably be painful.

I was working as a developer at Unisys (or it might have still been Burroughs), down the hall from the OS developers (for one of Burroughs' 3 distinct mainframe product lines, each with its own cpu architecture, operating system, system software, etc.). At that time, our mainframes could be patched while running, without re-booting; re-booting was considerably slower than patching. As a result, as each new OS patch was developed, it was applied on the fly to the machines in our dev environment, and they hadn't been rebooted for something like six months. It turned out that it was possible to have a sequence of patches, which when applied on the fly to a running system caused no problems, except that when it came time to reboot, it would crash. You can imagine the fun of trying to debug which change (or combination of several changes), out of many months of development work, had broken the boot process! These dev machines were shared with other teams (compiler developers, database engine developers, etc.), so the productivity hit waiting for our dev machines to be available again was substantial. After that, the policy was that the OS developers took turns coming in early one morning a week to reboot the dev machines, so in the worst case they'd only have to track down which of the last week's worth of patches might have broken the boot process.

Re: Cloud Server Reboots

#17

Post by tomjen3 »

Well that is a service I won't use then.

First of all you communicate all details so I know how I will be affected. Secondly you don't shut down my service ever, for any reason, other than lack of payment.

If you can't do those things you don't get to claim to have excellent support.

Re: Cloud Server Reboots

#18

Post by csmitheu »

chrissnell wrote: post #2

The timing of this announcement stinks. 2130 PDT on a Friday night, long after most folks have gone home. Making it even more painful, Rackspace is providing 1 hour advance notification. For those of us hosted in the US, there's a rolling reboot window that starts at 0400 PDT on Sunday morning. So, if you're a Rackspace customer and care that your app shuts down cleanly and restarts properly, you get to wake up at 04…

Sorry but in this case your application should have been tested to withstand all possible failure scenarios and restart scenarios so I have little sympathy. It genuinely sounds like you don't trust it to come back up which is not something I could sleep on. Things fail, sometimes violently.

Even with our company which has in in-house ops team and our own colocated kit, we expect process failures and restarts and plan accordingly. No sleep is lost even when something explodes at 2AM (other than for the ops team, who in this case you contracted out).

Re: Cloud Server Reboots

#19

Post by tomjen3 »

patio11 wrote: post #3

Those of you who are good at sysadminning don't need this advice, but for the fellow people who are only borderline competent in the room, pay particular attention to this reboot if you recently did "apt-get update" or similar to take care of the bash problem. I've shot myself in the foot before and accepted new updates to e.g. mysql that caused the existing config file to raise a hard error on load, which was only d…

This isn't to handle the bash problem?

What is it to fix then?

Re: Cloud Server Reboots

#20

Post by notatoad »

tomjen3 wrote: post #19
patio11 wrote: post #3

Those of you who are good at sysadminning don't need this advice, but for the fellow people who are only borderline competent in the room, pay particular attention to this reboot if you recently did "apt-get update" or similar to take care of the bash problem. I've shot myself in the foot before and accepted new updates to e.g. mysql that caused the existing config file to raise a hard error on load, which was only d…

This isn't to handle the bash problem? What is it to fix then?

nobody is saying yet, but probably the xen issue that's scheduled for disclosure on october 1
Post reply on HN