Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloud Server Reboots

status.rackspace.com

Post reply on HN
Print view View original

31–40 of 43 posts

Re: Cloud Server Reboots

#31

Post by kryptiskt »

csmitheu wrote: post #18
chrissnell wrote: post #2

The timing of this announcement stinks. 2130 PDT on a Friday night, long after most folks have gone home. Making it even more painful, Rackspace is providing 1 hour advance notification. For those of us hosted in the US, there's a rolling reboot window that starts at 0400 PDT on Sunday morning. So, if you're a Rackspace customer and care that your app shuts down cleanly and restarts properly, you get to wake up at 04…

Sorry but in this case your application should have been tested to withstand all possible failure scenarios and restart scenarios so I have little sympathy. It genuinely sounds like you don't trust it to come back up which is not something I could sleep on. Things fail, sometimes violently. Even with our company which has in in-house ops team and our own colocated kit, we expect process failures and restarts and plan…

> Sorry but in this case your application should have been tested to withstand all possible failure scenarios and restart scenarios so I have little sympathy.

Someone who thinks they can test all possible failure scenarios is severely lacking in imagination.

Re: Cloud Server Reboots

#32

Post by patio11 »

mikegioia wrote: post #26
patio11 wrote: post #3

Those of you who are good at sysadminning don't need this advice, but for the fellow people who are only borderline competent in the room, pay particular attention to this reboot if you recently did "apt-get update" or similar to take care of the bash problem. I've shot myself in the foot before and accepted new updates to e.g. mysql that caused the existing config file to raise a hard error on load, which was only d…

pay particular attention to this reboot if you recently did "apt-get update" or similar to take care of the bash problem I don't understand. apt-get update won't install new software on reboot. That just updates the index in apt. There's no other way to update bash without "apt-get update && apt-get install bash" and that won't impact anything on reboot. Your updated mysql config file sounds like you apt-get upgrade'…

Good thing I've I never totally hosed myself when picking the wrong word/letter in a command line as opposed to a non-executable HN comment.

cries softly to self

Re: Cloud Server Reboots

#33

Post by tsuraan »

praseodym wrote: post #28

And this is why live migrations (VMware vMotion, but also done by Google Compute Engine) are so awesome. Migrate VMs from server X to server Y, then patch and reboot server X. No VM downtime.

Yeah, Xen supports live migration as well (at least, libvirt supports live migration of xen backends: http://libvirt.org/migration.html). I would be interested in hearing why the big providers aren't apparently using it.

Re: Cloud Server Reboots

#34

Post by rmc »

tomjen3 wrote: post #19
patio11 wrote: post #3

Those of you who are good at sysadminning don't need this advice, but for the fellow people who are only borderline competent in the room, pay particular attention to this reboot if you recently did "apt-get update" or similar to take care of the bash problem. I've shot myself in the foot before and accepted new updates to e.g. mysql that caused the existing config file to raise a hard error on load, which was only d…

This isn't to handle the bash problem? What is it to fix then?

There is a bug in Xen, which hasn't been publically annouced yet.

Re: Cloud Server Reboots

#35

Post by rsync »

thaumaturgy wrote: post #14
jey wrote: post #7

Earlier quoted context omitted.

Sounds like it'd be a good idea to perform a deliberate reboot during off-peak hours and preemptively fix any issues.

You would think so (and I won't say it isn't), but if you're just one guy handling a few servers for your hosted SaaS, it's easy to fall into the "don't disturb the dust" trap -- especially once you've had a routine maintenance reboot or upgrade completely bone you. Not too long ago I had a 36-hour straight marathon sysadmin session when a routine update happened to break my particular stack without advance notice on…

Don't disturb the dust is actually a fairly good heuristic.

It sure would have helped those guys working at Chernobyl.

Re: Cloud Server Reboots

#36

Post by [deleted] »

csmitheu wrote: post #18
chrissnell wrote: post #2

The timing of this announcement stinks. 2130 PDT on a Friday night, long after most folks have gone home. Making it even more painful, Rackspace is providing 1 hour advance notification. For those of us hosted in the US, there's a rolling reboot window that starts at 0400 PDT on Sunday morning. So, if you're a Rackspace customer and care that your app shuts down cleanly and restarts properly, you get to wake up at 04…

Sorry but in this case your application should have been tested to withstand all possible failure scenarios and restart scenarios so I have little sympathy. It genuinely sounds like you don't trust it to come back up which is not something I could sleep on. Things fail, sometimes violently. Even with our company which has in in-house ops team and our own colocated kit, we expect process failures and restarts and plan…

[deleted]

Re: Cloud Server Reboots

#37

Post by [deleted] »

tomjen3 wrote: post #17

Well that is a service I won't use then. First of all you communicate all details so I know how I will be affected. Secondly you don't shut down my service ever, for any reason, other than lack of payment. If you can't do those things you don't get to claim to have excellent support.

[deleted]

Re: Cloud Server Reboots

#38

Post by csmitheu »

kryptiskt wrote: post #31
csmitheu wrote: post #18

Earlier quoted context omitted.

Sorry but in this case your application should have been tested to withstand all possible failure scenarios and restart scenarios so I have little sympathy. It genuinely sounds like you don't trust it to come back up which is not something I could sleep on. Things fail, sometimes violently. Even with our company which has in in-house ops team and our own colocated kit, we expect process failures and restarts and plan…

> Sorry but in this case your application should have been tested to withstand all possible failure scenarios and restart scenarios so I have little sympathy. Someone who thinks they can test all possible failure scenarios is severely lacking in imagination.

Sorry I should have used the word probable which is limited to power, network, software and storage failures. The solution to these is redundancy and to test each failure condition in succession and combinations thereof.

Sure there are more possibilities, but these should all be covered with a complete fallback DR strategy i.e. if not identifiable cause X, Y or Z then assume the worse, snapshot everything for debugging later and fallback to a complete restore from scratch.

The trick is to make X, Y and Z self-healing and part of your architecture provisioning.

I get the feeling most people here haven't dealt with large deployments and multiple site redundancy, five-nines reliability requirements and extensive DR planning, because the above is pretty obvious to those of us who have.

Re: Cloud Server Reboots

#39

Post by csmitheu »

mikegioia wrote: post #25
csmitheu wrote: post #18

Earlier quoted context omitted.

Sorry but in this case your application should have been tested to withstand all possible failure scenarios and restart scenarios so I have little sympathy. It genuinely sounds like you don't trust it to come back up which is not something I could sleep on. Things fail, sometimes violently. Even with our company which has in in-house ops team and our own colocated kit, we expect process failures and restarts and plan…

You're missing his point, and it's condescending/naive of you to assume that his app hasn't been tested to withstand it with no information to tell you that. My app has been thoroughly tested to withstand reboots; however, I have 22 machines with rackspace and (a) I've never tested all 22 going down at random times, and (b) I STILL want to be awake in case anything happens unexpectedly. The pain here is the timing an…

Well when you contract out your infrastructure that really should be part of your DR plan. Analysing failure conditions is really priority one when you put your stuff on someone else's turf as you have little control over this, despite contracts etc. You did do a DR plan right and did check your contract with RackSpace?

I want to be woken up if something doesn't come back, not if it does or even if it has gone into limp mode.

Seriously you're looking at the wrong end of the problem.

Re: Cloud Server Reboots

#40

Post by csmitheu »

csmitheu wrote: post #38
kryptiskt wrote: post #31

Earlier quoted context omitted.

> Sorry but in this case your application should have been tested to withstand all possible failure scenarios and restart scenarios so I have little sympathy. Someone who thinks they can test all possible failure scenarios is severely lacking in imagination.

Sorry I should have used the word probable which is limited to power, network, software and storage failures. The solution to these is redundancy and to test each failure condition in succession and combinations thereof. Sure there are more possibilities, but these should all be covered with a complete fallback DR strategy i.e. if not identifiable cause X, Y or Z then assume the worse, snapshot everything for debuggi…

Well it appears that 20 years' experience doesn't trump some total amateurs with a cloud to play with...
Post reply on HN