Batch Processing Millions and Millions of Images • Hacker News Board

Re: Batch Processing Millions and Millions of Images

#21

Post by patio11 » Sat, Jul 10, 2010, 8:16 PM UTC

If you want to make the "pick which setting is better" optimization a wee bit better, it is about two hours of work to rig up "Which do you like: image A, image B, or 'They're the same'?" and then simulated annealing your way to victory. Randomize which side you place the "better" setting on, obviously.

I'd almost be tempted to spend a few more hours and hook it up to mechanical turk rather than actually doing the classification myself. (Partially because I get bored easily and partially because I have all the artistic appreciation of a mole rat.)

Re: Batch Processing Millions and Millions of Images

#22

Post by tange » Sun, Jul 11, 2010, 10:40 AM UTC

lars512 wrote: post #14

The problem of resizing all these images is an "embarrassingly parallel" one, right? You don't care about how fast any individual image is resized, only how fast they're resized in aggregate, and each image is a nice small chunk of work. The author spends time tuning the number of workers and the number of OpenMP processes per worker for GraphicsMagick on his 16-core machines. Isn't this type of tuning a waste of tim…

Like this (if you have GNU Parallel http://www.gnu.org/software/parallel/ installed) to run on multiple computers:

find images | parallel -j+0 –trc {.}.out -S server1,server2,: “do_stuff {} > {.}.out”

Watch the intro video for GNU Parallel: http://www.youtube.com/watch?v=OpaiGYxkSuQ

Re: Batch Processing Millions and Millions of Images

#23

Post by mleonhard » Sun, Jul 11, 2010, 3:25 PM UTC

gaius wrote: post #18

mahmud wrote: post #15

Earlier quoted context omitted.

You really nailed it. Yes, yes and yes! This is the sort of stupid batch task that can be done with a shell-script using all the machines in the office after hours. Cut out the middle ware, cut down the threads and fancy IPC, and hunker down with good ole minimal number of processes on a bunch of machines. Heck, they might even see a big win if they use a different "server" memory management algorithm that's more tai…

Barely even a shell script; you could do it in a one-liner. Let's say you had 16000 images and 16 cores, $ cat filelist.txt|xargs -n 1000 -P 16 ./myconvertprog I do "batch" compression like this all the time.

Thanks for pointing out the '-P' argument for xargs!