How we found a bug in Amazon ELB

#1

Post by davideschiera » Thu, Apr 28, 2016, 4:44 PM UTC

How we found a bug in Amazon ELB
sysdig.com

Re: How we found a bug in Amazon ELB

#2

Post by jlgaddis » Thu, Apr 28, 2016, 5:15 PM UTC

As a network engineer, I'm constantly having to prove that "it's not the network" so I love reading others' technical analyses of similar things. Great troubleshooting and technical detail in this write-up.

Re: How we found a bug in Amazon ELB

#3

Post by jdreaver » Thu, Apr 28, 2016, 5:55 PM UTC

Hmm, this seems like a pretty big bug in connection draining. I feel like one instance per AZ is a pretty common scenario. Great article!

Re: How we found a bug in Amazon ELB

#4

Post by gighi » Thu, Apr 28, 2016, 6:07 PM UTC

jdreaver wrote: post #3

Hmm, this seems like a pretty big bug in connection draining. I feel like one instance per AZ is a pretty common scenario. Great article!

To be fair, the scenario is less common due to the fact that it happens just when the drained connections are terminated in a certain pattern (as shown in the charts). Still definitely common enough that can be easily replicated and cause real troubles :)

Re: How we found a bug in Amazon ELB

#5

Post by azundo » Thu, Apr 28, 2016, 6:16 PM UTC

We were told ELBs are explicitly not designed for long-running connections when we ran into this exact same issue so know that you will always be working around this design constraint if you do long-running connections through ELBs.

There's another case that the article doesn't really discuss (though the evidence of it is in the beginning when all connections drop simultaneously) where the ELB nodes themselves scale vertically at a particular threshold. I believe the setup described is still vulnerable to those scaling events.

Re: How we found a bug in Amazon ELB

#6

Post by gighi » Thu, Apr 28, 2016, 6:24 PM UTC

azundo wrote: post #5

We were told ELBs are explicitly not designed for long-running connections when we ran into this exact same issue so know that you will always be working around this design constraint if you do long-running connections through ELBs. There's another case that the article doesn't really discuss (though the evidence of it is in the beginning when all connections drop simultaneously) where the ELB nodes themselves scale…

We definitely observed such drops that we attributed to presumably internal ELB scaling activity, but they happen so occasionally that for the moment they haven't been a real issue, as opposed to this one described in the article which happened consistently at every deployment in our test environment.

Re: How we found a bug in Amazon ELB

#7

Post by DanielDent » Thu, Apr 28, 2016, 6:29 PM UTC

When the cloud work as desired, life is grand.

But when it doesn't, debugging might actually be simpler with less black boxes between you and the metal.

Re: How we found a bug in Amazon ELB

#8

Post by azundo » Thu, Apr 28, 2016, 6:32 PM UTC

gighi wrote: post #6

azundo wrote: post #5

We were told ELBs are explicitly not designed for long-running connections when we ran into this exact same issue so know that you will always be working around this design constraint if you do long-running connections through ELBs. There's another case that the article doesn't really discuss (though the evidence of it is in the beginning when all connections drop simultaneously) where the ELB nodes themselves scale…

We definitely observed such drops that we attributed to presumably internal ELB scaling activity, but they happen so occasionally that for the moment they haven't been a real issue, as opposed to this one described in the article which happened consistently at every deployment in our test environment.

Yeah, we've decided to live with the internal ELB scaling risks for the moment as well. We had the exact same situation where a deployment without gradual connection draining (even if we kept an instance in service in every AZ) would cause the ELBs to scale and drop all of our connections every time once we were at a certain scale. Definitely caused us a fair amount of confusion as it would happen minutes after the deploy when everything seemed to be calmed down again.

Re: How we found a bug in Amazon ELB

#9

Post by djb_hackernews » Thu, Apr 28, 2016, 6:42 PM UTC

In general, if you are using ELBs you should have at least 2 instances per AZ or cross zone load balancing enabled. I've seen this get teams several times.

The other thing to consider when deploying to the cloud with load balancers is to use an immutable architecture. Taking hosts out of service, updating them, and putting them back in service is a bit cumbersome at best and leaves you vulnerable to service outages.

Re: How we found a bug in Amazon ELB

#10

Post by bhz » Thu, Apr 28, 2016, 6:48 PM UTC

We experienced something like this a long while ago, something like 4-5 years. We still employ our workaround, which is to have a tiny "keepalive" instance in each AZ in the ELB.