Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nginx: a caching, thumbnailing, reverse proxying image server

charlesleifer.com

Post reply on HN
Print view View original

31–40 of 59 posts

Re: Nginx: a caching, thumbnailing, reverse proxying image server

#31

Post by mrb »

nly wrote: post #29
mrb wrote: post #19

His setup is vulnerable to hash length extension attacks: secure_link_md5 "my awesome secret $uri"; This means anyone can extend the URI (eg. adding "/../../../some/other/file" at the end of the filename) and compute a valid key, and force the server to access an arbitrary image file on the backend. If the backend is a private server not publicly accessible, then this may be a security issue. This is why in the ngx_h…

Length extension attacks work for both append and prepend. I don't know if you're right, or whether or not either of them are vulnerable, but the correct solution to this problem is to use the HMAC[0] construction. [0] https://en.wikipedia.org/wiki/Hash-based_message_authenticat...

Yes some other attacks can be performed if the secret is appended (we don't call them "length extensions attacks"), however they do not apply in this guy's case because the attacker does not control the URI that is hashed.

But yes, when in doubt, and to future-proof your code, it is good practice to use an HMAC anyway. Or use SHA-3 which is not vulnerable to any attack when prepending/appending a secret.

Re: Nginx: a caching, thumbnailing, reverse proxying image server

#33

Post by rgbrenner »

skrebbel wrote: post #3

If you use a CDN such as Cloudflare, you can do a similar trick but forget about the proxy cache (and associated storage needs) - as long as every image at every size gets a unique URL and you control http cache headers correctly, Cloudflare will store copies of every image size ever requested across their CDN network. In terms of this blog post, you'll only need the "resizing server". This is a very simple and state…

That's not quite true. Cloudflare (like virtually every CDN) only caches files at the edge server that handled the request. And they purge the files after a short period of time--even if your cache-control headers ask for a longer time. Most won't tell you how short (including Cloudflare), but some CDNs will delete files after as little as two hours.. 24 hours is very common. Most CDNs also give you no visibility about what is cached and where.

In Cloudflare's case, they operate 76 data centers. So every "short period of time" x up to 76, your server will need to regenerate each image.

So while a CDN is useful, a proxy cache can still be helpful.

(None of this applies to NuevoCloud though.. NuevoCloud has a global cache, with dedicated caches for each customer. So it's entirely possible to keep an image at the edge indefinitely)

Re: Nginx: a caching, thumbnailing, reverse proxying image server

#34

Post by coleifer »

mrb wrote: post #19

His setup is vulnerable to hash length extension attacks: secure_link_md5 "my awesome secret $uri"; This means anyone can extend the URI (eg. adding "/../../../some/other/file" at the end of the filename) and compute a valid key, and force the server to access an arbitrary image file on the backend. If the backend is a private server not publicly accessible, then this may be a security issue. This is why in the ngx_h…

Thanks, I've updated my post to append the secret key.

Re: Nginx: a caching, thumbnailing, reverse proxying image server

#35

Post by coleifer »

merlincorey wrote: post #8

This is a cool little project showcasing some of NginX's lesser known modules in action. However, I have one nitpick. The caching server's main location is a bit of a cargo-cult unoptimization. They have specified: location ^/(.+)$ { When the equivalent: location / { Is shorter, clearer, and does not incur a regular expression capture on every request that is never used.

Thank you, I've updated the post.

Re: Nginx: a caching, thumbnailing, reverse proxying image server

#36

Post by obelisk_ »

reipahb wrote: post #17
merlincorey wrote: post #8

This is a cool little project showcasing some of NginX's lesser known modules in action. However, I have one nitpick. The caching server's main location is a bit of a cargo-cult unoptimization. They have specified: location ^/(.+)$ { When the equivalent: location / { Is shorter, clearer, and does not incur a regular expression capture on every request that is never used.

The difference is that the last one also matches requests to "/", while the regex ensures that there is at least a single character after the "/". That allows "/" to be used for other purposes, e.g. redirecting to the main page of the site, or displaying a nice error page. (However, in this case it simply shows the default Debian Nginx installation page: http://m.charlesleifer.com/ )

Actually, Nginx are way ahead of you. Here's the way to handle that.

    location = / { [ configuration A ] }

    location / { [ configuration B ] }
To learn more about this, I recommend http://nginx.org/en/docs/http/ngx_http_core_module.html

The Nginx docs are not always great but they are far more correct than most blog posts about Nginx so that is why I have turned to use only the official docs when I have a question about how to do something with Nginx.

Re: Nginx: a caching, thumbnailing, reverse proxying image server

#37

Post by coleifer »

frankwiles wrote: post #28
dwightgunning wrote: post #16

Earlier quoted context omitted.

Perhaps better a better question is why you were using Varnish in the first place? I'm genuinely curious because I've never stood up Varnish before so maybe there's an anti-pattern / gotcha to learn from.

Well without some sort of caching, his site very well could fall over when being on the front page of something like HN. Putting Varnish in front a site, even with something simple like a 1 minute cache on everything, makes you pretty much immune from having "large amounts of traffic" being a real problem.

Exactly, thanks frank! :P

Re: Nginx: a caching, thumbnailing, reverse proxying image server

#38

Post by coleifer »

jhgg wrote: post #26

We use a similar set-up at work, but we don't do the image resizing in nginx, rather we have a python backend that it proxies up to, that runs a tiny wsgi app that uses requests & pil to do the resizing and transforming. It uses proxy_cache_lock to dedupe requests to the backend.

If all you need is crop or resize, might as well use nginx in my opinion.

Re: Nginx: a caching, thumbnailing, reverse proxying image server

#39

Post by jhgg »

coleifer wrote: post #38
jhgg wrote: post #26

We use a similar set-up at work, but we don't do the image resizing in nginx, rather we have a python backend that it proxies up to, that runs a tiny wsgi app that uses requests & pil to do the resizing and transforming. It uses proxy_cache_lock to dedupe requests to the backend.

If all you need is crop or resize, might as well use nginx in my opinion.

There was one annoyance we have to fix that I don't think the nginx module supports & that's rotating images based on their exif orientation (mobile devices love to upload landscape images w/ an exif-orientation in portrait).

Python is definitely an OK solution for this as it turns out. We resize a few million of images a day on-demand on two n1-highcpu-8 GCE instances. Although, they could easily be a fourth the size, as CPU generally peaks out at 20-25% during peak hours.

We've actually tried more specialized services for this, like sharp + a threadpool using node.js and it actually performed terribly. As it turns out, blocking the gunicorn event loop by processing image resizes created a good amount of back-pressure to load balance incoming requests to each worker process.

Post reply on HN