OpenRoss – fast, scalable, on-demand image resizer
developers.lyst.com
OpenRoss – fast, scalable, on-demand image resizer
1–10 of 34 posts
Re: OpenRoss – fast, scalable, on-demand image resizer
#2As a healthy feedback - the correct architecture sits right below your nose, since you already have all of the components.
You should pre-process images in your scraping cycles, and not when a client comes to request it.
In this way, your "scale" is always predefined, bounded, expected, and much smaller - defined by your scraping scale and not user scale.
Good luck!
Re: OpenRoss – fast, scalable, on-demand image resizer
#3As a healthy feedback - the correct architecture sits right below your nose, since you already have all of the components. You should pre-process images in your scraping cycles, and not when a client comes to request it. In this way, your "scale" is always predefined, bounded, expected, and much smaller - defined by your scraping scale and not user scale. Good luck!
Isn't that what they started with, and decided against? "In our infancy, we saved all product images with 10 preset sizes, and then rendered the image which was nearest in size to what we required. As we grew, this solution became unwieldy for the levels of traffic we were experiencing and nor was it appropriate for our mobile app."
Re: OpenRoss – fast, scalable, on-demand image resizer
#4At scale, processing thousands of highly granular items (images) in any way is an embarrassingly parallel task even in its most crude and naive implementation.
So claiming "fast", ok, but claiming "scalable" feels like a redundant buzzword, a bit hand-wavey.
When you make such generic claims, be quick to explain what you mean, or people will not treat you seriously.
Re: OpenRoss – fast, scalable, on-demand image resizer
#5Why not skip the backend completely and just use nginx + cdn?
(Have nginx proxy_pass to S3 and image_filter the response)
http://nginx.org/en/docs/http/ngx_http_image_filter_module.h...
Re: OpenRoss – fast, scalable, on-demand image resizer
#6As a healthy feedback - the correct architecture sits right below your nose, since you already have all of the components. You should pre-process images in your scraping cycles, and not when a client comes to request it. In this way, your "scale" is always predefined, bounded, expected, and much smaller - defined by your scraping scale and not user scale. Good luck!
Isn't that what they started with, and decided against? "In our infancy, we saved all product images with 10 preset sizes, and then rendered the image which was nearest in size to what we required. As we grew, this solution became unwieldy for the levels of traffic we were experiencing and nor was it appropriate for our mobile app."
They say that, but there's one giant missing ", because..." in that paragraph.
They never explain why they render in 10 sizes (why don't they know which sizes they need), and why is loading one of those sizes on a mobile app not feasible.
It's stated like that and left hanging in the air.
Either their use case is very weird, or they're deliberately vague for some reason only they know about.
Also to clarify why 10 sizes is weird, normally what you'd do is double the width/height of your images with every size (so quadruple the pixels), same as one does with mip-maps. Or with icons.
Let's say the smallest sensible size is 128x128 (minimum needed to discern a product, and easy to downsize from there, won't get much smaller at good quality).
So we have 128, 256, 512, 1024, 2048, that's 5 sizes. And I'm pretty sure the images they need aren't over 2048x.
So 10 sizes is just pointless.
And with OpenRoss, doing cropping and adding whitespace at the server, producing pointless image duplicates, is senseless. Even the most crippled client-side technique can do the cropping and whitespace for you (yeah, even html).
This screams "we didn't think this through much".
Re: OpenRoss – fast, scalable, on-demand image resizer
#7At scale, processing thousands of highly granular items (images) in any way is an embarrassingly parallel task even in its most crude and naive implementation. So claiming "fast", ok, but claiming "scalable" feels like a redundant buzzword, a bit hand-wavey. When you make such generic claims, be quick to explain what you mean, or people will not treat you seriously.
Yeah what i was thinking. There is no state to be managed. Its just a pipeline which doesnt not anything more than more machines to handles more load. Its relatively easy to implement.
Re: OpenRoss – fast, scalable, on-demand image resizer
#8Earlier quoted context omitted.
Isn't that what they started with, and decided against? "In our infancy, we saved all product images with 10 preset sizes, and then rendered the image which was nearest in size to what we required. As we grew, this solution became unwieldy for the levels of traffic we were experiencing and nor was it appropriate for our mobile app."
They say that, but there's one giant missing ", because..." in that paragraph. They never explain why they render in 10 sizes (why don't they know which sizes they need), and why is loading one of those sizes on a mobile app not feasible. It's stated like that and left hanging in the air. Either their use case is very weird, or they're deliberately vague for some reason only they know about. Also to clarify why 10 si…
It's a chicken/egg scenario. How does the resource cost of resizing on-demand compare to storing all sizes? Also, like @mantraxC said, are all those image sizes really needed?
I have a system I'm trying to sunset that has a 160 and 130 size. Totally redundant, and doesn't really save that much space/bandwidth.
Still, OpenRoss is a cool project to learn from. It might not fit many use-cases, but apparently it works for Lyst.
Re: OpenRoss – fast, scalable, on-demand image resizer
#9Earlier quoted context omitted.
Isn't that what they started with, and decided against? "In our infancy, we saved all product images with 10 preset sizes, and then rendered the image which was nearest in size to what we required. As we grew, this solution became unwieldy for the levels of traffic we were experiencing and nor was it appropriate for our mobile app."
They say that, but there's one giant missing ", because..." in that paragraph. They never explain why they render in 10 sizes (why don't they know which sizes they need), and why is loading one of those sizes on a mobile app not feasible. It's stated like that and left hanging in the air. Either their use case is very weird, or they're deliberately vague for some reason only they know about. Also to clarify why 10 si…
When the site was originally designed, there were only 4 or 5 sizes, one for the feed, one for the product page, one for related products, one for thumbnails, etc.
As the design changed, extra sizes got added to the image processing, and as with most early stage start-ups, it worked therefore there was no need to fix it.
We eventually ended up in the position of having 2.5M products all with preprocessed images of certain sizes from our design history. As we wanted more flexibility with design, but also knew that a lot of our images would have a very low likelihood of being accessed (fashion items have single runs and are never remade in future seasons), a big batch process didn't seem appropriate. Additionally, it would mean storing several different copies of images in our S3 bucket, even if we knew the product would not likely be seen again.
A more attractive solution (at least to us) was to do the hybrid approach, where we would resize on demand, and then cache for a long time. This way, we only do the processing for images that need it, in almost a functionally identical way to large scale batch processing, but the process is demand-led.
Re: OpenRoss – fast, scalable, on-demand image resizer
#10Why not skip the backend completely and just use nginx + cdn? (Have nginx proxy_pass to S3 and image_filter the response) http://nginx.org/en/docs/http/ngx_http_image_filter_module.h...
The CDN does not have all sizes of images. We use the ondemand resizer+cache because we may modify our website design in the future and need a new image size. Serving the exact image size makes our pages faster to render and saves bandwidth. Plus that image filter is quite limited and doesn't handle compositing.