They're still taking downtime for this... Even if they're forced to have a no-VIP no-HA no-LB setup (seems insane to me) it should be much simpler to set the DNS TTL to a low value right before and switch it to the new IP after the new box comes up.
Don’t run this on any system you expect to be up they said, but we did it anyway
11–20 of 24 posts
Re: Don’t run this on any system you expect to be up they said, but we did it anyway
#12So... they're not running any kind of devops system at all. If they were, they could just run the upgrade on the image, test it, deploy it. All the "months of careful planning and many many tests" they did are basically wasted time. I wouldn't be proud of this, quite the opposite. I would suggest critically reviewing the entire infrastructure management strategy since months lost to a single upgrade is obviously indi…
I work on a "devops" team and this is similar to what we do. We have a "reinstall the whole damn image" flag but using it by default would just cause unnecessary downtime for us, especially in our non-production environments.
Re: Don’t run this on any system you expect to be up they said, but we did it anyway
#13They're still taking downtime for this... Even if they're forced to have a no-VIP no-HA no-LB setup (seems insane to me) it should be much simpler to set the DNS TTL to a low value right before and switch it to the new IP after the new box comes up.
Yes, this is a wonderful technical hack, but terrible as a business strategy. They are admitting that they don't have a good architecture that would make upgrades easy.
Re: Don’t run this on any system you expect to be up they said, but we did it anyway
#14A nice writeup of a neat (if risky) upgrade. > static IPs FWIW, I personally love Virtual IPs (VIPs) for this (basically, an existing network interface advertises serving more than one IP, and can change that IP dynamically between servers with an arp call). The downside is that there are a lot of cloud providers who don't support externally available VIPs. They do, however, offer their own nearly-identical solution…
That's addressed in the article: "We purposely don’t employ dynamic IPs to retain multi-cloud deployment capabilities and prevent vendor lock-in with one platform."
I'd hazard a guess that the cost of moving IPs would severely outweigh the cost of switching VIP implementations, in terms of preventing a move across providers.
Re: Don’t run this on any system you expect to be up they said, but we did it anyway
#15A nice writeup of a neat (if risky) upgrade. > static IPs FWIW, I personally love Virtual IPs (VIPs) for this (basically, an existing network interface advertises serving more than one IP, and can change that IP dynamically between servers with an arp call). The downside is that there are a lot of cloud providers who don't support externally available VIPs. They do, however, offer their own nearly-identical solution…
That's addressed in the article: "We purposely don’t employ dynamic IPs to retain multi-cloud deployment capabilities and prevent vendor lock-in with one platform."
That still makes no sense. AWS and DigitalOcean both allow for static IP addresses that can be migrated between instances. There's no reason for crazy in-place upgrades.
Re: Don’t run this on any system you expect to be up they said, but we did it anyway
#16They're still taking downtime for this... Even if they're forced to have a no-VIP no-HA no-LB setup (seems insane to me) it should be much simpler to set the DNS TTL to a low value right before and switch it to the new IP after the new box comes up.
That assumes they have control over the DNS. Sounds like they don't with many end customers.
Re: Don’t run this on any system you expect to be up they said, but we did it anyway
#17They're still taking downtime for this... Even if they're forced to have a no-VIP no-HA no-LB setup (seems insane to me) it should be much simpler to set the DNS TTL to a low value right before and switch it to the new IP after the new box comes up.
That assumes they have control over the DNS. Sounds like they don't with many end customers.
They shouldn't be giving their customers a static IP, they should be giving them a 'customername.ourplatform.com' address that the customer can point a CNAME at.
Re: Don’t run this on any system you expect to be up they said, but we did it anyway
#18So... they're not running any kind of devops system at all. If they were, they could just run the upgrade on the image, test it, deploy it. All the "months of careful planning and many many tests" they did are basically wasted time. I wouldn't be proud of this, quite the opposite. I would suggest critically reviewing the entire infrastructure management strategy since months lost to a single upgrade is obviously indi…
It sounds to me like they are in fact running a giant devops system, all for the purpose of not using virtual static IPs. Instead of just provisioning fresh VMs and migrating customer data they're doing this massive upgrade in place on existing machines to avoid losing the assigned IPs. I guess they decided the benefits of being cloud provider agnostic outweighed the downside of spending months of man hours automatin…
A good system can handle multiple requirements. You present a false dichotomy.
Re: Don’t run this on any system you expect to be up they said, but we did it anyway
#19A nice writeup of a neat (if risky) upgrade. > static IPs FWIW, I personally love Virtual IPs (VIPs) for this (basically, an existing network interface advertises serving more than one IP, and can change that IP dynamically between servers with an arp call). The downside is that there are a lot of cloud providers who don't support externally available VIPs. They do, however, offer their own nearly-identical solution…
That's addressed in the article: "We purposely don’t employ dynamic IPs to retain multi-cloud deployment capabilities and prevent vendor lock-in with one platform."
Why not have the machines presented to the clients or other interfaces by a virtual IP from an application delivery controller such as F5's BigIP or similar? And then remove the dependency on static NIC's on a virtual appliance?
Seems counter intuitive to run virtual appliances on static addresses if it can be avoided.
Re: Don’t run this on any system you expect to be up they said, but we did it anyway
#20I think the authors lost a good opportunity to move towards containers to avoid these problems in the future. While interesting academically, is wrong for the long run