Batch Processing Millions and Millions of Images
codeascraft.etsy.com
Batch Processing Millions and Millions of Images
1–10 of 23 posts
Re: Batch Processing Millions and Millions of Images
#2Several years ago I had to do something similar (10 of thousands instead of millions though).
Our problem was that we were making a composite image but sometimes the 'background' image content was shifted to the left or right. (e.g. All images were the same size with a white background but some were made from different source template and so the actual content sometimes started at pixel 25 and other times at pixel 35.) To make it worse they were all JPEGS.
I ended up writing a program to find the bounding box of the content (with a fudge factor to account for the JPEG dithering) and identified the bad images to be fixed by hand. It was a nice diversion from SQL and ASP.
Re: Batch Processing Millions and Millions of Images
#3Several years ago I had to do something similar (10 of thousands instead of millions though). Our problem was that we were making a composite image but sometimes the 'background' image content was shifted to the left or right. (e.g. All images were the same size with a white background but some were made from different source template and so the actual content sometimes started at pixel 25 and other times at pixel 35…
What tools did you use and how long did it take?
Re: Batch Processing Millions and Millions of Images
#4Several years ago I had to do something similar (10 of thousands instead of millions though). Our problem was that we were making a composite image but sometimes the 'background' image content was shifted to the left or right. (e.g. All images were the same size with a white background but some were made from different source template and so the actual content sometimes started at pixel 25 and other times at pixel 35…
What tools did you use and how long did it take?
This was in 2002 and I used Python with the PIL module.
It didn't really take all that long, less than a day for the whole project. Probably 2-3 hours of run time on my desktop.
I wasn't actually doing anything but identifying the bad images so I wasn't pounding the disk with alternating reads/writes.
Re: Batch Processing Millions and Millions of Images
#5Earlier quoted context omitted.
What tools did you use and how long did it take?
This was in 2002 and I used Python with the PIL module. It didn't really take all that long, less than a day for the whole project. Probably 2-3 hours of run time on my desktop. I wasn't actually doing anything but identifying the bad images so I wasn't pounding the disk with alternating reads/writes.
PIL is fucking amazing, I've never seen anything like it for any other language. Last year I used it to write a computer vision utility for doing black-box UI testing on embedded medical devices, and I can't imagine having used anything else. Right now I'm fighting RMagick on another project, I would have set the building on fire if I'd had to deal with it's bullshit on the CV project.
It links directly with libjpeg, libpng, etc. instead of via some imagemagick bullshit. Instead of being fucked if what you want to do doesn't line up with an existing magick command, it gives you the tools to do it yourself, and even pretty performantly since it can give you numpy arrays.
Re: Batch Processing Millions and Millions of Images
#6Good, detailed post. I just did about 250K images using ImageMagick and a shell script. It was across 3 boxes and I noticed quite a performance difference between an older and newer version of IM. It could have been another factor, I suppose.
One thing I'm skeptical about:
"We found out, almost by accident, that using the previously down-sized “large” images resulted in better quality and faster processing than starting with the original full-size images."
Obviously, the processing speed will be faster with a down-sized image, but I can't see how starting with a smaller image will give you better quality. Unless the original image was so large that downsizing caused a lot of detail loss (in which case the image should have been sharpened).
Re: Batch Processing Millions and Millions of Images
#7Did you investigate resizing them on the fly with some kind of caching layer? As one of the posters notes, it seems likely that a lot of those images will be very rarely seen, if ever.
Re: Batch Processing Millions and Millions of Images
#8Call me naive, but I think for any serious processing, you need to dig into the actual underlying algorithm and implementation to made difference. The hardware difference between generations are huge, different arch can have a big impact in term of performance (really down to the detail: cache-line, bandwidth, and SIMD inst, image processing is sensitive to all these above). Tuning your custom implementation can suddenly be worthwhile.
Re: Batch Processing Millions and Millions of Images
#9Good, detailed post. I just did about 250K images using ImageMagick and a shell script. It was across 3 boxes and I noticed quite a performance difference between an older and newer version of IM. It could have been another factor, I suppose. One thing I'm skeptical about: "We found out, almost by accident, that using the previously down-sized “large” images resulted in better quality and faster processing than start…
If you use a wrong algorithm, that can be a case. For example, when you use bilinear interpolation to "down-size" image, for large image, you can end up interpolating between two far away pixels whereas if you use a smaller image, the two pixels you use are already averaged (interpolated), thus, can get a more pleasing result.
Re: Batch Processing Millions and Millions of Images
#10Why did you do this as a batch job? Do you run batch jobs instead of resizing them when a user uploads the pictures? Thanks for sharing.