Looks like Nvidia is doing a proper content marketing push to compete against AMD.
Facebook Trains ImageNet in 1 Hour
11–20 of 50 posts
Re: Facebook Trains ImageNet in 1 Hour
#12relevant paper from facebook https://research.fb.com/publications/ImageNet1kIn1h/
This is the URL that jonbaer originally submitted (sadly at an awkward time). I meant to send mail about it (there's no "Really! I vouch for this!" for low point stories that languish), but I see the result worked out anyway.
Re: Facebook Trains ImageNet in 1 Hour
#13tl;dr they found a clever way to spread the training across 256 GPUs by synchronising the stochastic gradient descent
this seems like the trivial, most obvious way to parallelize training across GPUs. Not, imo, clever.
The important bit here is that they've shown that large mini batch sizes still can maintain accuracy if you slow the learning rate.
Re: Facebook Trains ImageNet in 1 Hour
#14I'd always conceptualized decreasing batch size as a performance/memory optimization to deal with the fact that datasets don't all fit into memory and to reduce overall training time. You look at batch_size samples and compute the sum of the gradient of the errors to update the network weights so as to reduce the error -- shouldn't a larger batch_size inherently provide more information about the optimal direction of…
[Warning: far from an expert here] No, batches also help you escape local minima.
[Another non-expert commenting]
Your comment didn't make sense to me at first, but I think I get it now. Even if you were able to fit the entire dataset into memory, batches are still a good idea, because optimizing on the entire dataset is non-convex and will likely lead you into a local minimum. However, what is a local minimum for one batch may not be a local minimum for the next batch, which helps you escape.
This explains why optimization gets harder with very large batch sizes - the gradients for different batches become more similar (as they resemble the "global" gradient more closely), so you become more susceptible to local minima. I think this also explains why the learning rate scaling helps - it increases the variance across gradients, and helps you escape local minima.
Re: Facebook Trains ImageNet in 1 Hour
#15I can't seem to find it anywhere but what is the interconnect between servers being used? NVLink is used internally for GPU to GPU communication within a single box...correct? But this sounds like it takes a cluster of 32 of their 8 GPU Big Basin boxes.
Something similar to Infiniband
Re: Facebook Trains ImageNet in 1 Hour
#16Some observations:
* for synchronous model-based distributed training to scale linearly, the time required to broadcast the model must be much larger than the time required for a worker (GPU) to process a batch
* it's not strict synchronous training, as when gradients are computed at a worker, they are transmitted to all workers - so the driver doesn't have to send models to all 32 workers at the same time (8 GPUs per worker makes 256 GPUs in total).
* there are extremely large batch sizes (8196)
* it's a good network (50 Gb Ethernet, albeit not infiniband)
So, the relative amount of work done training at each worker is much higher than the time spent broadcasting the model (which is quite small (~100 MB, i think)) to the workers for each iteration. For larger models with smaller batch sizes, this relationship would break down. The interesting contribution here is that you can have massive batch sizes and Facebook provided a heuristic for adjusting the learning rate to converge with such massive batch sizes.
Re: Facebook Trains ImageNet in 1 Hour
#17I'd always conceptualized decreasing batch size as a performance/memory optimization to deal with the fact that datasets don't all fit into memory and to reduce overall training time. You look at batch_size samples and compute the sum of the gradient of the errors to update the network weights so as to reduce the error -- shouldn't a larger batch_size inherently provide more information about the optimal direction of…
One of the better articles I found about this topic when I've been learning for my exam:
http://sebastianruder.com/optimizing-gradient-descent/index....
Re: Facebook Trains ImageNet in 1 Hour
#18tl;dr they found a clever way to spread the training across 256 GPUs by synchronising the stochastic gradient descent
this seems like the trivial, most obvious way to parallelize training across GPUs. Not, imo, clever. The important bit here is that they've shown that large mini batch sizes still can maintain accuracy if you slow the learning rate.
very personally, clever means "well, that would have taken me a long time to figure out and code properly"
Re: Facebook Trains ImageNet in 1 Hour
#19Earlier quoted context omitted.
[Warning: far from an expert here] No, batches also help you escape local minima.
[Another non-expert commenting] Your comment didn't make sense to me at first, but I think I get it now. Even if you were able to fit the entire dataset into memory, batches are still a good idea, because optimizing on the entire dataset is non-convex and will likely lead you into a local minimum. However, what is a local minimum for one batch may not be a local minimum for the next batch, which helps you escape. Thi…
This explanation of why larger batches might make optimization harder makes a fair amount of sense to me. This comment on Quora helps a bit as well - with a slightly different approach to characterizing the trade off of batch size/accuracy/runtime performance.
https://www.quora.com/Intuitively-how-does-mini-batch-size-a...
I wonder if rather than computing a single gradient for a large batch you could simultaneously compute a gradient for the batch and for several subsets of the batch -- then pick or combine the gradient subset(s) that most differ from the full batch result. Not sure if that would work out to a computational efficiency gain.
Are there any optimizers that dynamically scale the batch size up/down based on an online metric?
Re: Facebook Trains ImageNet in 1 Hour
#20How much time would it have taken using Torch instead of Caffe2? (I still don't understand which I should use...)
It depends on your application. For most general purposes caffe2 will be fine. For research and pushing the limits pytorch is your best bet.