Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we found a bug in Amazon ELB

sysdig.com

Post reply on HN
Print view View original

21–30 of 43 posts

Re: How we found a bug in Amazon ELB

#21

Post by tedmiston »

kirbypineapple wrote: post #20
nzoschke wrote: post #15

Great article. The Sysdig team really knows how to root cause tough problems. The Sysdig tools can be invaluable for getting and making sense of low level data. If you want to play with ELBs, rolling deploys, connection draining to ECS containers, I humbly submit the open source Convox project I am working on. https://github.com/convox/rack It sets up a peer reviewed, production tested batteries-included VPC, ECS, AS…

Do you have any thoughts on how to scale load balancers horizontally and on demand? I've played briefly with attempting some dynamic DNS routing based on health checks to re-route traffic from balancers that have been shut down due to low traffic, but DNS really isn't designed to work this way.

I'm not clear what you're asking... Do you mean auto scaling the EC2 instances in the load balancer? Or auto scaling the # of load balancers? Or something else?

Of course the former is very common with Auto Scaling Groups [1] [2]. Then you can use round robin or session sticky routing algorithms in the load balancers.

(Apologies if I'm totally off-base for what you were asking.)

1: http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide...

2: http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide...

Re: How we found a bug in Amazon ELB

#22

Post by jasonwatkinspdx »

kirbypineapple wrote: post #20
nzoschke wrote: post #15

Great article. The Sysdig team really knows how to root cause tough problems. The Sysdig tools can be invaluable for getting and making sense of low level data. If you want to play with ELBs, rolling deploys, connection draining to ECS containers, I humbly submit the open source Convox project I am working on. https://github.com/convox/rack It sets up a peer reviewed, production tested batteries-included VPC, ECS, AS…

Do you have any thoughts on how to scale load balancers horizontally and on demand? I've played briefly with attempting some dynamic DNS routing based on health checks to re-route traffic from balancers that have been shut down due to low traffic, but DNS really isn't designed to work this way.

DNS based load balancing has a lot of caveats due to how clients can hold onto stale information. To do better means stepping up to running anycast via BGP, a signifigant jump in complexity. That's why services like Cloudflare exist.

Re: How we found a bug in Amazon ELB

#23

Post by faceplanter »

djb_hackernews wrote: post #9

In general, if you are using ELBs you should have at least 2 instances per AZ or cross zone load balancing enabled. I've seen this get teams several times. The other thing to consider when deploying to the cloud with load balancers is to use an immutable architecture. Taking hosts out of service, updating them, and putting them back in service is a bit cumbersome at best and leaves you vulnerable to service outages.

While I agree with having an immutable arch is preferable but in some cases it's not viable. In one of our projects we re-use the instances like in the article since we deploy multiple times an hour. In AWS you are billed for each started hour which in this case would mean that we would pay a lot extra if we created new instances for each deploy.

Re: How we found a bug in Amazon ELB

#24

Post by djb_hackernews »

faceplanter wrote: post #23
djb_hackernews wrote: post #9

In general, if you are using ELBs you should have at least 2 instances per AZ or cross zone load balancing enabled. I've seen this get teams several times. The other thing to consider when deploying to the cloud with load balancers is to use an immutable architecture. Taking hosts out of service, updating them, and putting them back in service is a bit cumbersome at best and leaves you vulnerable to service outages.

While I agree with having an immutable arch is preferable but in some cases it's not viable. In one of our projects we re-use the instances like in the article since we deploy multiple times an hour. In AWS you are billed for each started hour which in this case would mean that we would pay a lot extra if we created new instances for each deploy.

I do wish AWS had more granular EC2 billing, and I expect that to come soon since GCE offers it. But 2 things:

1) If you are at the scale of deploying several times an hour, the instance hour cost would probably look like a rounding error for your entire AWS spend, I'd imagine.

2) At that cadence you'll definitely benefit from using containers and a container scheduler (Kube, ECS, etc). Reuse the infrastructure but redeploy your apps to your hearts content.

Re: How we found a bug in Amazon ELB

#25

Post by itaifrenkel »

djb_hackernews wrote: post #9

In general, if you are using ELBs you should have at least 2 instances per AZ or cross zone load balancing enabled. I've seen this get teams several times. The other thing to consider when deploying to the cloud with load balancers is to use an immutable architecture. Taking hosts out of service, updating them, and putting them back in service is a bit cumbersome at best and leaves you vulnerable to service outages.

I've heard that cross zone load balancing means the vpc encryption does not cover traffic between zones (the traffic is isolated like in ec2 classic). Is that substantiated?

Re: How we found a bug in Amazon ELB

#27

Post by otterley »

itaifrenkel wrote: post #25
djb_hackernews wrote: post #9

In general, if you are using ELBs you should have at least 2 instances per AZ or cross zone load balancing enabled. I've seen this get teams several times. The other thing to consider when deploying to the cloud with load balancers is to use an immutable architecture. Taking hosts out of service, updating them, and putting them back in service is a bit cumbersome at best and leaves you vulnerable to service outages.

I've heard that cross zone load balancing means the vpc encryption does not cover traffic between zones (the traffic is isolated like in ec2 classic). Is that substantiated?

Network communication between instances in a VPC is not encrypted, and never has been, to my knowledge. Perhaps you're thinking of VPN?

Re: How we found a bug in Amazon ELB

#28

Post by narsil »

We recently discovered that the NAT Gateway also terminates connections by issuing a RST packet when it receives the next packet for a connection that it believes to have timed out, effectively causing the new request to fail. The previous recommended approach of NATing in VPC was to use NAT instances, which sent FIN packets when the timeout was hit, cleanly closing the connection. That behavior was far better, since it indicated that a new request should re-connect first.

AWS Support indicated that this was a feature of the new NAT Gateways, even though it breaks outbound connections made by popular implementations such as the Requests python library's urllib3 connection pools. This is pretty unfortunate, and has been a roadblock in migrating to the NAT Gateways.

Re: How we found a bug in Amazon ELB

#29

Post by gleenn »

azundo wrote: post #8
gighi wrote: post #6

Earlier quoted context omitted.

We definitely observed such drops that we attributed to presumably internal ELB scaling activity, but they happen so occasionally that for the moment they haven't been a real issue, as opposed to this one described in the article which happened consistently at every deployment in our test environment.

Yeah, we've decided to live with the internal ELB scaling risks for the moment as well. We had the exact same situation where a deployment without gradual connection draining (even if we kept an instance in service in every AZ) would cause the ELBs to scale and drop all of our connections every time once we were at a certain scale. Definitely caused us a fair amount of confusion as it would happen minutes after the d…

The author said he needed at least 2 instances in a AZ to avoid the bug, and used that as his workaround in the mean time that Amazon works on the bug.

Re: How we found a bug in Amazon ELB

#30

Post by MBCook »

faceplanter wrote: post #23
djb_hackernews wrote: post #9

In general, if you are using ELBs you should have at least 2 instances per AZ or cross zone load balancing enabled. I've seen this get teams several times. The other thing to consider when deploying to the cloud with load balancers is to use an immutable architecture. Taking hosts out of service, updating them, and putting them back in service is a bit cumbersome at best and leaves you vulnerable to service outages.

While I agree with having an immutable arch is preferable but in some cases it's not viable. In one of our projects we re-use the instances like in the article since we deploy multiple times an hour. In AWS you are billed for each started hour which in this case would mean that we would pay a lot extra if we created new instances for each deploy.

Is Elastic Beanstalk not an option? It doesn't replace hosts on redeploy so you wouldn't end up cycling through unnecessary instance.
Post reply on HN