Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nginx with dynamic upstreams

tenzer.dk

Post reply on HN
Print view View original

21–30 of 55 posts

Re: Nginx with dynamic upstreams

#21

Post by Tenzer »

jpgvm wrote: post #20
Tenzer wrote: post #19

Earlier quoted context omitted.

You can also use Nginx for the PROXY protocol, that was added in version 1.5.12. You should just add "proxy_protocol" to your "listen" directive: http://nginx.org/en/docs/http/ngx_http_core_module.html#list... .

Cool, does it allow you to control the balancing to enforce source-ip -> backend mapping? This is required due to the nature of Websockets UPGRADE and most semi-stateful Websockets servers.

Sure, that's not really related to the protocol used. It's instead handled by the upstream module and the "ip_hash" flag: http://nginx.org/en/docs/http/ngx_http_upstream_module.html#....

Re: Nginx with dynamic upstreams

#22

Post by devonkim »

lisper wrote: post #14
twic wrote: post #11

Earlier quoted context omitted.

It's not quite as bad as that. AWS's short TTL means that they can change the IP whenever they like, not that they will . I would imagine that most of the time, long-lived connections to an ELB will be fine. If the ELB does get a new address, then yes, the connection will fail, the client will have to reconnect, and when it does, it will need to do a fresh address lookup. But since the connection is over a network, f…

The problem is not the frequency with which it happens. The problem is that the way you get notified that it has happened is that IP packets suddenly start to get delivered to the wrong machine. If you're lucky, the net result will simply be a dropped TCP connection. If you're not lucky, pretty much arbitrarily bad things can happen. Now, it is true that IP is not reliable, and so arbitrarily bad things can happen at…

I like to think of the problem as an inherent issue with cache invalidation in distributed systems that has no straightforward solution in an Internet architecture. DNS problems greatly resemble what it's like to have dangling pointers, for another analogy.

I think the expected solution approach is a little like validating 2FA logins. To account for system variability, you accept that transactions can take up to a certain time and you allow multiple values (the similarities probably end there, admittedly). With a more advanced solution you would even advise clients to invalidate their own cache too (similar to load shedding by advising / redirecting clients - you can see this in 2FA logins sometimes where you're asked for the sequence after the once-valid key just entered). So I think you'd need to accept multiple prior CNAME resolutions to account for longer lived transactions and make sure each entry change will be valid only for so long. Being able to be notified of these changes programmatically would be really nice though specifically for AWS. Perhaps AWS Lambda or SNS could be leveraged for pushing AWS DNS state change notifications to your system?

One approach I've seen some folks do is to simply reload their nginx configurations across their backend nodes to refresh the cached entry. It's probably intractable without sacrificing a lot of theoretical availability with in a degenerate state of reloading upon each request when your CNAME changes several times a second. Some of my colleagues have experienced problems even with the nginx proxy_pass directive that most people say is the most recommended free solution and also recommended in the article.

Re: Nginx with dynamic upstreams

#23

Post by bmurphy1976 »

I spent a long time fighting this battle with nginx. I eventually came to the conclusion that while it was technically possible, it just wasn't worth the trouble (and yes, I tried various alternatives such as Tengine and OpenResty).

What I settled on was the following:

Ngninx for SSL termination in front of an haproxy load balancer. I wrote a simple Python script (92 lines of code) that spits out a new haproxy configuration and gracefully reloads haproxy whenever the DNS changes. That has (after fixing a bug or two in my code) been FAR more reliable than any other solution to this problem we've tried.

Bonus: haproxy is a superior load balancer with better runtime metrics and health checking options than nginx.

Re: Nginx with dynamic upstreams

#24

Post by bmurphy1976 »

Tenzer wrote: post #12
lisper wrote: post #9

Earlier quoted context omitted.

It's worse than a bug, it's a Really Bad Design (tm). Bugs are usually unintentional, but this is a deliberate reinvention of a wheel (IP address caching) that doesn't need to be reinvented. But this is not just nginx's fault. It's also Bad Design on the part of AWS because switching IP addresses this way means you can't keep a TCP socket open to an ELB machine for more than 60 seconds at a time because you never kno…

Regarding how the ELB works, then I think the AWS engineers have a different idea on how to implement stuff. AWS is very much a dynamic platform and the engineers seem to have embraced this when they came up with this solution. It doesn't necessarily mean you can't use websockets through an ELB, it just means that you would need to be able to handle reconnects, but that shouldn't be a new challenge for any system rel…

My analysis shows an AWS ELB changes ip addresses on us roughly every 2 weeks. Often enough to cause problems if you aren't prepared but infrequent enough to give you a false confidence that things are working as designed.

Re: Nginx with dynamic upstreams

#25

Post by adamonduty »

bmurphy1976 wrote: post #23

I spent a long time fighting this battle with nginx. I eventually came to the conclusion that while it was technically possible, it just wasn't worth the trouble (and yes, I tried various alternatives such as Tengine and OpenResty). What I settled on was the following: Ngninx for SSL termination in front of an haproxy load balancer. I wrote a simple Python script (92 lines of code) that spits out a new haproxy config…

What part of the battle couldn't you solve? Doesn't the author's setting of the resolver directive work for you, or is there something I'm missing?

Also, the author doesn't mention it, but you can also set a custom cache timeout with the valid=Xs option.

http://nginx.org/en/docs/http/ngx_http_core_module.html#reso...

Re: Nginx with dynamic upstreams

#26

Post by sprobertson »

Tenzer wrote: post #3
corobo wrote: post #2

Thank you for this, it's a decent start for me personally. I was previously using the DNS director in Varnish 3.x to do this which was removed in 4.x. The longer time goes on the more risk it is to be running older software so this has been a great help. Oddly enough I'd never really considered nginx for the job despite using it to reverse proxy elsewhere. Sometimes you just need a poke in the right direction :) Now…

It's good to hear it was of use to somebody :) I know you can use a map in Nginx to do what you ask for, as long as you have a list of domains already: http://nginx.org/en/docs/http/ngx_http_map_module.html#map . I can only imagine it also being possible to make fully dynamic, I just don't have a clear way of doing it in mind right now.

Lua + Redis is a nice way to do this, I have a small project https://github.com/spro/simon that does dynamic routing and load balancing based on Hostname -> IP:Port sets in Redis. Adding a new route is as simple as:

  sadd backends:hnapi.dev 192.168.0.42:10555

Re: Nginx with dynamic upstreams

#27

Post by MichaelGG »

jon-wood wrote: post #8
jpmattia wrote: post #7

Am I wrong for thinking that failure to honor the TTL in the first place is a bug?

I find it particularly grating that the official way to deal with Nginx ignoring the advertised TTL is to pay $1500 per year, per server, for an Nginx Plus license.

Yes. Some of the nginx plus features are very small, and there's even 3rd party modules that add such features. This particular case seems like an outright bug, with a hack workaround. Would they take a patch to fix it?

It makes me think that they must be intentionally omitting features in order to make plus valuable. That seems rather cheesy. Obviously it's their code and right, but I think it shows how hard it is to profit off an open source program.

Re: Nginx with dynamic upstreams

#28

Post by ishigoemon »

lisper wrote: post #16
Tenzer wrote: post #12

Earlier quoted context omitted.

Regarding how the ELB works, then I think the AWS engineers have a different idea on how to implement stuff. AWS is very much a dynamic platform and the engineers seem to have embraced this when they came up with this solution. It doesn't necessarily mean you can't use websockets through an ELB, it just means that you would need to be able to handle reconnects, but that shouldn't be a new challenge for any system rel…

> you would need to be able to handle reconnects, but that shouldn't be a new challenge for any system relying on connections being open for long That's true, but it kind of misses the point. Normally, if a connection is dropped it means something has gone wrong. In this case, connections are dropped by design , and there is no way (AFAICT) to work around this. Designing so that behavior that is otherwise the result…

I'd argue that this design promotes building reliable applications. A system that cannot reconnect is fragile, and the best way to know if the system can handle that failure is to occasionally induce the event. Assuming that you are running across a lossless, ideal network is, IMHO, the very definition of Bad Design.

"If it hurts, do it more often." -- Martin Fowler

Re: Nginx with dynamic upstreams

#30

Post by justincormack »

Tenzer wrote: post #10
jpmattia wrote: post #7

Am I wrong for thinking that failure to honor the TTL in the first place is a bug?

I think the reasoning for it is for performance. If you can make all the DNS queries you need before you start serving any requests, then you don't have to wait for DNS servers while clients hammer your server. Ideally I would have like it to be an option though, instead of it basically having become a feature in Nginx Plus - if it wasn't for the way I described in the post.

It should just schedule a DNS update before the previous one has expired, and if the addresses have changed shift over, so it should not cause any delays.
Post reply on HN