Great work done!
How we found a bug in Amazon ELB
31–40 of 43 posts
Re: How we found a bug in Amazon ELB
#32As a network engineer, I'm constantly having to prove that "it's not the network" so I love reading others' technical analyses of similar things. Great troubleshooting and technical detail in this write-up.
What are some examples where you proved this? Curious about scenarios...
Re: How we found a bug in Amazon ELB
#33Earlier quoted context omitted.
While I agree with having an immutable arch is preferable but in some cases it's not viable. In one of our projects we re-use the instances like in the article since we deploy multiple times an hour. In AWS you are billed for each started hour which in this case would mean that we would pay a lot extra if we created new instances for each deploy.
Is Elastic Beanstalk not an option? It doesn't replace hosts on redeploy so you wouldn't end up cycling through unnecessary instance.
Genuine question: Why is it okay to reuse instances because it's controlled via an abstraction layer, as opposed to doing it yourself?
Re: How we found a bug in Amazon ELB
#34This is a great article to read. The author mentions WireShark - fun fact: the founder of Sysdig, Loris, is also the creator of WireShark.
He created WinPcap, not Wireshark.
Re: How we found a bug in Amazon ELB
#35We were told ELBs are explicitly not designed for long-running connections when we ran into this exact same issue so know that you will always be working around this design constraint if you do long-running connections through ELBs. There's another case that the article doesn't really discuss (though the evidence of it is in the beginning when all connections drop simultaneously) where the ELB nodes themselves scale…
Really interesting. Only two weeks ago we've been told by an AWS Architect that "if you need persistent TCP connections to servers avoid the ELB and connect straight with your [scaled] EC2 instances".
This was for a higher load scenario though.
Re: How we found a bug in Amazon ELB
#36Earlier quoted context omitted.
Is Elastic Beanstalk not an option? It doesn't replace hosts on redeploy so you wouldn't end up cycling through unnecessary instance.
Genuine question: Why is it okay to reuse instances because it's controlled via an abstraction layer, as opposed to doing it yourself?
I agree, that doesn't make a difference.
I only mentioned EB because it does that kind of thing for you and if you don't have a highly complicated setup it makes rolling updates without changing instances very easy.
Re: How we found a bug in Amazon ELB
#37We recently discovered that the NAT Gateway also terminates connections by issuing a RST packet when it receives the next packet for a connection that it believes to have timed out, effectively causing the new request to fail. The previous recommended approach of NATing in VPC was to use NAT instances, which sent FIN packets when the timeout was hit, cleanly closing the connection. That behavior was far better, since…
Full disclosure: I'm an engineer at AWS and I work on NAT Gateway :)
Thanks for the pointer to urllib3 - we'll take a look at it and see if there's anything we can do about the behavior. One of the challenges with sending "FIN" on timeout is, as you write ... it closes the connections cleanly.
Some TCP based protocols (Including even HTTP in some modes) use a successful connection close to indicate that an object has been transferred fully; so what we've seen is that a network connection may stall (internet packet loss for example) ... then the connection eventually times out ... and the "FIN" falsely conveys that the entire object has been transferred. The end result is a truncated object, which is no good either.
Re: How we found a bug in Amazon ELB
#38Earlier quoted context omitted.
Do you have any thoughts on how to scale load balancers horizontally and on demand? I've played briefly with attempting some dynamic DNS routing based on health checks to re-route traffic from balancers that have been shut down due to low traffic, but DNS really isn't designed to work this way.
I'm not clear what you're asking... Do you mean auto scaling the EC2 instances in the load balancer? Or auto scaling the # of load balancers? Or something else? Of course the former is very common with Auto Scaling Groups [1] [2]. Then you can use round robin or session sticky routing algorithms in the load balancers. (Apologies if I'm totally off-base for what you were asking.) 1: http://docs.aws.amazon.com/AutoScal…
I meant the latter; if your load balancers are overwhelmed how do you scale them? Further to that point, is it possible to create an architecture where load balancers are responsive and can spin up in response to traffic? If you have to deal with loads that are prone to bursts, you need to allocate those load balancers in such a way that they can handle the worst case scenario.
Re: How we found a bug in Amazon ELB
#39Earlier quoted context omitted.
I'm not clear what you're asking... Do you mean auto scaling the EC2 instances in the load balancer? Or auto scaling the # of load balancers? Or something else? Of course the former is very common with Auto Scaling Groups [1] [2]. Then you can use round robin or session sticky routing algorithms in the load balancers. (Apologies if I'm totally off-base for what you were asking.) 1: http://docs.aws.amazon.com/AutoScal…
I meant the latter; if your load balancers are overwhelmed how do you scale them? Further to that point, is it possible to create an architecture where load balancers are responsive and can spin up in response to traffic? If you have to deal with loads that are prone to bursts, you need to allocate those load balancers in such a way that they can handle the worst case scenario.
I haven't worked at the scale where dynamically scaling the load balancers themselves is the bottleneck. I think you pose an interesting question, and I'm hoping someone with more knowledge can comment on that.
Re: How we found a bug in Amazon ELB
#40Earlier quoted context omitted.
I meant the latter; if your load balancers are overwhelmed how do you scale them? Further to that point, is it possible to create an architecture where load balancers are responsive and can spin up in response to traffic? If you have to deal with loads that are prone to bursts, you need to allocate those load balancers in such a way that they can handle the worst case scenario.
I haven't worked at the scale where dynamically scaling the load balancers themselves is the bottleneck. I think you pose an interesting question, and I'm hoping someone with more knowledge can comment on that.
I don't believe there is any way to scale out the ELB from user side; you can contact AWS support to 'pre warm' ELB's for high traffic sites before cutting over DNS to them.