Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nginx with dynamic upstreams

tenzer.dk

Post reply on HN
Print view View original

11–20 of 55 posts

Re: Nginx with dynamic upstreams

#11

Post by twic »

lisper wrote: post #9
jpmattia wrote: post #7

Am I wrong for thinking that failure to honor the TTL in the first place is a bug?

It's worse than a bug, it's a Really Bad Design (tm). Bugs are usually unintentional, but this is a deliberate reinvention of a wheel (IP address caching) that doesn't need to be reinvented. But this is not just nginx's fault. It's also Bad Design on the part of AWS because switching IP addresses this way means you can't keep a TCP socket open to an ELB machine for more than 60 seconds at a time because you never kno…

It's not quite as bad as that. AWS's short TTL means that they can change the IP whenever they like, not that they will. I would imagine that most of the time, long-lived connections to an ELB will be fine.

If the ELB does get a new address, then yes, the connection will fail, the client will have to reconnect, and when it does, it will need to do a fresh address lookup. But since the connection is over a network, failure is a possibility regardless of what AWS does, and so clients need to be able to detect failure and reconnect anyway.

Re: Nginx with dynamic upstreams

#12

Post by Tenzer »

lisper wrote: post #9
jpmattia wrote: post #7

Am I wrong for thinking that failure to honor the TTL in the first place is a bug?

It's worse than a bug, it's a Really Bad Design (tm). Bugs are usually unintentional, but this is a deliberate reinvention of a wheel (IP address caching) that doesn't need to be reinvented. But this is not just nginx's fault. It's also Bad Design on the part of AWS because switching IP addresses this way means you can't keep a TCP socket open to an ELB machine for more than 60 seconds at a time because you never kno…

Regarding how the ELB works, then I think the AWS engineers have a different idea on how to implement stuff. AWS is very much a dynamic platform and the engineers seem to have embraced this when they came up with this solution.

It doesn't necessarily mean you can't use websockets through an ELB, it just means that you would need to be able to handle reconnects, but that shouldn't be a new challenge for any system relying on connections being open for long. Also, the load balancer servers doesn't switch every 60 seconds, you can have connections running for a lot longer than that. I would also assume the load balancers keep handling connections for a while after they were taken out of the DNS rotation, in order to make sure DNS caches are updated before the IP addresses stops working.

Re: Nginx with dynamic upstreams

#13

Post by click170 »

Tenzer wrote: post #10
jpmattia wrote: post #7

Am I wrong for thinking that failure to honor the TTL in the first place is a bug?

I think the reasoning for it is for performance. If you can make all the DNS queries you need before you start serving any requests, then you don't have to wait for DNS servers while clients hammer your server. Ideally I would have like it to be an option though, instead of it basically having become a feature in Nginx Plus - if it wasn't for the way I described in the post.

I think there's room for interpretation but that in cases like this, it is indeed a bug.

Many applications are programmed this way but I firmly believe that an application needs to honor the TTL of a DNS request for any subsequent connections, for exactly this reason. DNS records change. Sometimes frequently. IMO you shouldn't need yo kick your apps to get them to use the new hostname for subsequent connections.

Re: Nginx with dynamic upstreams

#14

Post by lisper »

twic wrote: post #11
lisper wrote: post #9

Earlier quoted context omitted.

It's worse than a bug, it's a Really Bad Design (tm). Bugs are usually unintentional, but this is a deliberate reinvention of a wheel (IP address caching) that doesn't need to be reinvented. But this is not just nginx's fault. It's also Bad Design on the part of AWS because switching IP addresses this way means you can't keep a TCP socket open to an ELB machine for more than 60 seconds at a time because you never kno…

It's not quite as bad as that. AWS's short TTL means that they can change the IP whenever they like, not that they will . I would imagine that most of the time, long-lived connections to an ELB will be fine. If the ELB does get a new address, then yes, the connection will fail, the client will have to reconnect, and when it does, it will need to do a fresh address lookup. But since the connection is over a network, f…

The problem is not the frequency with which it happens. The problem is that the way you get notified that it has happened is that IP packets suddenly start to get delivered to the wrong machine. If you're lucky, the net result will simply be a dropped TCP connection. If you're not lucky, pretty much arbitrarily bad things can happen.

Now, it is true that IP is not reliable, and so arbitrarily bad things can happen at any time and you do have to be prepared for those. The problem here is that there is no notification. Potentially bad things are happening here not because something has gone wrong, but by design. That, IMHO, is the very definition of Bad Design.

Normally, when you make a DNS change, you control the DNS and the affected end points. That way you know when the change is happening, and you can give yourself as much time as you need to make the transition in an orderly way. With ELB, the only guarantee you have is the TTL. With a 60 second TTL, that means you have at most 60 seconds to do the transition, and even that is only if you notice it when it happens (and the only way to guarantee that is to poll the name server constantly).

Re: Nginx with dynamic upstreams

#15

Post by JimmyL »

Tenzer wrote: post #12
lisper wrote: post #9

Earlier quoted context omitted.

It's worse than a bug, it's a Really Bad Design (tm). Bugs are usually unintentional, but this is a deliberate reinvention of a wheel (IP address caching) that doesn't need to be reinvented. But this is not just nginx's fault. It's also Bad Design on the part of AWS because switching IP addresses this way means you can't keep a TCP socket open to an ELB machine for more than 60 seconds at a time because you never kno…

Regarding how the ELB works, then I think the AWS engineers have a different idea on how to implement stuff. AWS is very much a dynamic platform and the engineers seem to have embraced this when they came up with this solution. It doesn't necessarily mean you can't use websockets through an ELB, it just means that you would need to be able to handle reconnects, but that shouldn't be a new challenge for any system rel…

AWS support has said that ELBs will continue to accept connections (and use the correct backend) for at least an hour after the CNAME stops resolving to a particular IP.

Re: Nginx with dynamic upstreams

#16

Post by lisper »

Tenzer wrote: post #12
lisper wrote: post #9

Earlier quoted context omitted.

It's worse than a bug, it's a Really Bad Design (tm). Bugs are usually unintentional, but this is a deliberate reinvention of a wheel (IP address caching) that doesn't need to be reinvented. But this is not just nginx's fault. It's also Bad Design on the part of AWS because switching IP addresses this way means you can't keep a TCP socket open to an ELB machine for more than 60 seconds at a time because you never kno…

Regarding how the ELB works, then I think the AWS engineers have a different idea on how to implement stuff. AWS is very much a dynamic platform and the engineers seem to have embraced this when they came up with this solution. It doesn't necessarily mean you can't use websockets through an ELB, it just means that you would need to be able to handle reconnects, but that shouldn't be a new challenge for any system rel…

> you would need to be able to handle reconnects, but that shouldn't be a new challenge for any system relying on connections being open for long

That's true, but it kind of misses the point. Normally, if a connection is dropped it means something has gone wrong. In this case, connections are dropped by design, and there is no way (AFAICT) to work around this. Designing so that behavior that is otherwise the result of things going wrong is now the normal designed-for behavior is, IMHO, the very definition of Bad Design.

Re: Nginx with dynamic upstreams

#17

Post by lisper »

JimmyL wrote: post #15
Tenzer wrote: post #12

Earlier quoted context omitted.

Regarding how the ELB works, then I think the AWS engineers have a different idea on how to implement stuff. AWS is very much a dynamic platform and the engineers seem to have embraced this when they came up with this solution. It doesn't necessarily mean you can't use websockets through an ELB, it just means that you would need to be able to handle reconnects, but that shouldn't be a new challenge for any system rel…

AWS support has said that ELBs will continue to accept connections (and use the correct backend) for at least an hour after the CNAME stops resolving to a particular IP.

Ah, I didn't know that. That makes a big difference.

But there's still a significant hole here: is there any way to get notified that this has happened other than polling the DNS?

Re: Nginx with dynamic upstreams

#18

Post by jpgvm »

lisper wrote: post #9
jpmattia wrote: post #7

Am I wrong for thinking that failure to honor the TTL in the first place is a bug?

It's worse than a bug, it's a Really Bad Design (tm). Bugs are usually unintentional, but this is a deliberate reinvention of a wheel (IP address caching) that doesn't need to be reinvented. But this is not just nginx's fault. It's also Bad Design on the part of AWS because switching IP addresses this way means you can't keep a TCP socket open to an ELB machine for more than 60 seconds at a time because you never kno…

You can hold a TCP connection open through an ELB basically as long as you want. The default idle timeout is 60s but can be increased to 1hr, this is a non-factor if you are sending any sort of data though.

When the "routing rug" is pulled out from under you all you need to do is re-resolve and re-establish the TCP connection which will likely live on for days (in most cases weeks) without disconnecting again.

This is fine for most use cases I am aware of.

As for Websockets, you will need to run the ELB in TCP mode to do that and probably run a real HTTP proxy behind it that supports Websockets/UPGRADE and uses constant source-ip hashing and supports the TCP PROXY protocol. i.e HAProxy. You can run HAProxy or other any other proxy that matches the above in an ELB to get good highly available Websockets proxy layer.

Re: Nginx with dynamic upstreams

#19

Post by Tenzer »

jpgvm wrote: post #18
lisper wrote: post #9

Earlier quoted context omitted.

It's worse than a bug, it's a Really Bad Design (tm). Bugs are usually unintentional, but this is a deliberate reinvention of a wheel (IP address caching) that doesn't need to be reinvented. But this is not just nginx's fault. It's also Bad Design on the part of AWS because switching IP addresses this way means you can't keep a TCP socket open to an ELB machine for more than 60 seconds at a time because you never kno…

You can hold a TCP connection open through an ELB basically as long as you want. The default idle timeout is 60s but can be increased to 1hr, this is a non-factor if you are sending any sort of data though. When the "routing rug" is pulled out from under you all you need to do is re-resolve and re-establish the TCP connection which will likely live on for days (in most cases weeks) without disconnecting again. This i…

You can also use Nginx for the PROXY protocol, that was added in version 1.5.12. You should just add "proxy_protocol" to your "listen" directive: http://nginx.org/en/docs/http/ngx_http_core_module.html#list....

Re: Nginx with dynamic upstreams

#20

Post by jpgvm »

Tenzer wrote: post #19
jpgvm wrote: post #18

Earlier quoted context omitted.

You can hold a TCP connection open through an ELB basically as long as you want. The default idle timeout is 60s but can be increased to 1hr, this is a non-factor if you are sending any sort of data though. When the "routing rug" is pulled out from under you all you need to do is re-resolve and re-establish the TCP connection which will likely live on for days (in most cases weeks) without disconnecting again. This i…

You can also use Nginx for the PROXY protocol, that was added in version 1.5.12. You should just add "proxy_protocol" to your "listen" directive: http://nginx.org/en/docs/http/ngx_http_core_module.html#list... .

Cool, does it allow you to control the balancing to enforce source-ip -> backend mapping?

This is required due to the nature of Websockets UPGRADE and most semi-stateful Websockets servers.

Post reply on HN