I don't really see a benefit in updating existing instances in this manner. Launching replacement instances with the new code is much easier for us, and it also provides a super fast means of rollback.
How we found a bug in Amazon ELB
11–20 of 43 posts
Re: How we found a bug in Amazon ELB
#12I don't really see a benefit in updating existing instances in this manner. Launching replacement instances with the new code is much easier for us, and it also provides a super fast means of rollback.
Both approaches are reasonable (and there's also a third one, ship your application in containers and replace containers instead of instances).
We update existing instances because in our test environment we deploy at every single new commit (we absolutely love that), and we have hundreds (or more) a day. At that pace, replacing instances would be more time consuming (again, for our specific use case) and less cost efficient.
Plus, updating existing instances is handled automatically by AWS Code Deploy, which provides a very good deploying pipeline that you can control using the aws cli tool.
There are other minor advantages but those are the two main ones.
Re: How we found a bug in Amazon ELB
#13Somewhat unrelated to the ELB problem identified, but an alternative solution to the original deployment problem: assuming that the collectors are stateless (seem to be) start off the deployment by spinning up a new collector with the new code installed. Then, proceed with the deployment in the original fashion. Once that's over, kill the extra collector. This will ensure that load is distributed roughly in the same manner, over the same number of nodes during the deployment as before the deployment. Depending on load caused by initiating a connection, more than one extra mode may be utilized. In any case, this is a much simpler approach than baking in application-level connection termination. All for a few extra bucks per deploy and a small amount of engineering time up front.
Re: How we found a bug in Amazon ELB
#14I don't really see a benefit in updating existing instances in this manner. Launching replacement instances with the new code is much easier for us, and it also provides a super fast means of rollback.
One potential issue with spinning up new instances on EC2 is that for larger instance sizes, if you care about the instances being in a certain AZ, there may not be enough available to do this.
Re: How we found a bug in Amazon ELB
#15Great article. The Sysdig team really knows how to root cause tough problems. The Sysdig tools can be invaluable for getting and making sense of low level data.
If you want to play with ELBs, rolling deploys, connection draining to ECS containers, I humbly submit the open source Convox project I am working on.
https://github.com/convox/rack
It sets up a peer reviewed, production tested batteries-included VPC, ECS, ASG, ELB, etc cluster in minutes.
If the conclusion of this Sysdig post was that you always need to run 2 instances per AZ for the best reliability, I would strongly consider adding that knowledge into the tools either as a default or a production check.
Since it sounds like an ELB bug I'll keep the 3 instances in 3 AZs default.
Re: How we found a bug in Amazon ELB
#16I don't really see a benefit in updating existing instances in this manner. Launching replacement instances with the new code is much easier for us, and it also provides a super fast means of rollback.
It's really hard to replace the simplicity and reliability of letting ASG (and generally CloudFormation) roll out new instances.
However best practices always evolve...
I'd say that rolling out containers on ECS is starting to really show advantages.
It is now generally:
- easier to build and push an image than burn an AMI - faster to boot a container than an instance - faster to finish a deploy with options like min containers in service and a slack instance or two
To be honest most teams don't actually need the extra agility that containers promise.
But if I was starting an AWS setup from scratch I'd strongly consider containers on ECS.
In addition to the speed there is more portability with containers and a whole new generation of tools coming in the ecosystem.
Re: How we found a bug in Amazon ELB
#17I don't really see a benefit in updating existing instances in this manner. Launching replacement instances with the new code is much easier for us, and it also provides a super fast means of rollback.
Both approaches are reasonable (and there's also a third one, ship your application in containers and replace containers instead of instances). We update existing instances because in our test environment we deploy at every single new commit (we absolutely love that), and we have hundreds (or more) a day. At that pace, replacing instances would be more time consuming (again, for our specific use case) and less cost e…
That's an awesomely aggressive deployment rate and a great reason to do instance mutation.
Does something verify every commit in the testing environment too?
Re: How we found a bug in Amazon ELB
#18Earlier quoted context omitted.
Both approaches are reasonable (and there's also a third one, ship your application in containers and replace containers instead of instances). We update existing instances because in our test environment we deploy at every single new commit (we absolutely love that), and we have hundreds (or more) a day. At that pace, replacing instances would be more time consuming (again, for our specific use case) and less cost e…
That's an awesomely aggressive deployment rate and a great reason to do instance mutation. Does something verify every commit in the testing environment too?
Yes, every commit gets pulled by jenkins which builds the whole thing, runs unit tests and then starts the deployment once the tests pass.
Re: How we found a bug in Amazon ELB
#19Somewhat unrelated to the ELB problem identified, but an alternative solution to the original deployment problem: assuming that the collectors are stateless (seem to be) start off the deployment by spinning up a new collector with the new code installed. Then, proceed with the deployment in the original fashion. Once that's over, kill the extra collector. This will ensure that load is distributed roughly in the same…
Definitely a feasible approach. Let's just say that the reality has a bit more color and we have some other practical advantages in controlling the exact moment when we disconnect a particular client :)
Re: How we found a bug in Amazon ELB
#20Great article. The Sysdig team really knows how to root cause tough problems. The Sysdig tools can be invaluable for getting and making sense of low level data. If you want to play with ELBs, rolling deploys, connection draining to ECS containers, I humbly submit the open source Convox project I am working on. https://github.com/convox/rack It sets up a peer reviewed, production tested batteries-included VPC, ECS, AS…
Do you have any thoughts on how to scale load balancers horizontally and on demand? I've played briefly with attempting some dynamic DNS routing based on health checks to re-route traffic from balancers that have been shut down due to low traffic, but DNS really isn't designed to work this way.