Don’t run this on any system you expect to be up they said, but we did it anyway
1–10 of 24 posts
Re: Don’t run this on any system you expect to be up they said, but we did it anyway
#2A nice writeup of a neat (if risky) upgrade.
> static IPs
FWIW, I personally love Virtual IPs (VIPs) for this (basically, an existing network interface advertises serving more than one IP, and can change that IP dynamically between servers with an arp call). The downside is that there are a lot of cloud providers who don't support externally available VIPs. They do, however, offer their own nearly-identical solution (such as Elastic IPs from Amazon).
The use of VIPs or similar could have potentially avoided the need for such a risky upgrade, potentially also saving millions of dollars in the process. Of course, I could simply be missing some hidden requirement from customers that they couldn't use VIPs but that's pretty uncommon, even in the finance industry.
Re: Don’t run this on any system you expect to be up they said, but we did it anyway
#3Very nice!
Re: Don’t run this on any system you expect to be up they said, but we did it anyway
#4Very nice!
Re: Don’t run this on any system you expect to be up they said, but we did it anyway
#5Given how much memory some servers have these days, which for an application node is often more than the necessary hard-disk capacity, this is quite a clever approach.
Re: Don’t run this on any system you expect to be up they said, but we did it anyway
#6So... they're not running any kind of devops system at all. If they were, they could just run the upgrade on the image, test it, deploy it. All the "months of careful planning and many many tests" they did are basically wasted time.
I wouldn't be proud of this, quite the opposite. I would suggest critically reviewing the entire infrastructure management strategy since months lost to a single upgrade is obviously indicative of greater problems.
Re: Don’t run this on any system you expect to be up they said, but we did it anyway
#7So... they're not running any kind of devops system at all. If they were, they could just run the upgrade on the image, test it, deploy it. All the "months of careful planning and many many tests" they did are basically wasted time. I wouldn't be proud of this, quite the opposite. I would suggest critically reviewing the entire infrastructure management strategy since months lost to a single upgrade is obviously indi…
I'm not sure that I'm quite as critical, the technique itself looks useful for some scenarios.
I've worked on a Windows project where we solved a similar problem by booting from vhd, so you can "just" write a new vhd, uniquify it with the per machine config, update the boot menu and reboot - all data are on a separate volume, naturally
I'm surprised they went to all this effort for only 2000 machines though.
Re: Don’t run this on any system you expect to be up they said, but we did it anyway
#8A nice writeup of a neat (if risky) upgrade. > static IPs FWIW, I personally love Virtual IPs (VIPs) for this (basically, an existing network interface advertises serving more than one IP, and can change that IP dynamically between servers with an arp call). The downside is that there are a lot of cloud providers who don't support externally available VIPs. They do, however, offer their own nearly-identical solution…
That's addressed in the article: "We purposely don’t employ dynamic IPs to retain multi-cloud deployment capabilities and prevent vendor lock-in with one platform."
Re: Don’t run this on any system you expect to be up they said, but we did it anyway
#9So... they're not running any kind of devops system at all. If they were, they could just run the upgrade on the image, test it, deploy it. All the "months of careful planning and many many tests" they did are basically wasted time. I wouldn't be proud of this, quite the opposite. I would suggest critically reviewing the entire infrastructure management strategy since months lost to a single upgrade is obviously indi…
It sounds to me like they are in fact running a giant devops system, all for the purpose of not using virtual static IPs.
Instead of just provisioning fresh VMs and migrating customer data they're doing this massive upgrade in place on existing machines to avoid losing the assigned IPs.
I guess they decided the benefits of being cloud provider agnostic outweighed the downside of spending months of man hours automating in-place OS upgrades.
Re: Don’t run this on any system you expect to be up they said, but we did it anyway
#10A nice writeup of a neat (if risky) upgrade. > static IPs FWIW, I personally love Virtual IPs (VIPs) for this (basically, an existing network interface advertises serving more than one IP, and can change that IP dynamically between servers with an arp call). The downside is that there are a lot of cloud providers who don't support externally available VIPs. They do, however, offer their own nearly-identical solution…
That's addressed in the article: "We purposely don’t employ dynamic IPs to retain multi-cloud deployment capabilities and prevent vendor lock-in with one platform."
I was really confused by this. Cloud vendors are not bring-your-own-IP AFAIK, how can they even get a non-virtual static IP addresses on the cloud?