Batch Processing Millions and Millions of Images

This was for a site design after the images had been uploaded.

Nice article. FWIW Perl is often called the "Swiss army chain saw," not "Swiss army knife." Thanks for sharing your experience.

You really nailed it. Yes, yes and yes!

This is the sort of stupid batch task that can be done with a shell-script using all the machines in the office after hours. Cut out the middle ware, cut down the threads and fancy IPC, and hunker down with good ole minimal number of processes on a bunch of machines.

Heck, they might even see a big win if they use a different "server" memory management algorithm that's more tailored to batch, not-so-responsive applications.

Mmh

"In fact, the research phase of this project took longer than the batch processing itself.  That was clearly time well spent."

I'm not so sure about this. A sufficiently parallelized but otherwise unresearched script could have been created in a fraction of the time. So you would just generate the correct thumbnails on new uploads and then let the batch job run for, say, two weeks. That naive approach will take you around a days worth of development time.

The time you gained by doing it the naive way you then put into the rewrite of that legacy component that prevented you from on-the-fly generating the images.

What you have done here is, IMHO, wasted time in a legacy solution and you wasted hardware resources for storage of the additional pictures of which probably only a minority will actually be seen anyways.

Don't get me wrong: I'm sure you had a lot of fun and there's little more exciting things than seeing your code perform magnitudes quicker than the initial approach. But in the context of your quote, I have to disagree.

Unless there's more background you didn't tell us about.

Hate to be the Monday morning quarterback but hey, it's Saturday morning and I'm stuck in a Starbucks in Benicia. So, this is a nice walk through the optimization process but it is a fundamentally unscalable system. When the workload triples next year, does it make sense to scale up the hardware? (No)

This system doesn't scale out without ad-hoc partitioning. I think Etsy's approach here should have been informed by the New York Times' project a few years ago. They converted 4 TB of scanned tiffs (their article archive) into PDFs on a hadoop cluster running on ec2 and s3. Parallelizing the process across hardware nodes is the scale free way to do this, exactly what hadoop was intended for.

All that said, I know there are smart folks at Etsy. I'll give 'em the benefit of the doubt; there may be a good reason not to go that route but this write up didn't make that clear.