Facebook Trains ImageNet in 1 Hour
news.developer.nvidia.com
Facebook Trains ImageNet in 1 Hour
1–10 of 50 posts
Re: Facebook Trains ImageNet in 1 Hour
#2relevant paper from facebook https://research.fb.com/publications/ImageNet1kIn1h/
Re: Facebook Trains ImageNet in 1 Hour
#3I can't seem to find it anywhere but what is the interconnect between servers being used? NVLink is used internally for GPU to GPU communication within a single box...correct? But this sounds like it takes a cluster of 32 of their 8 GPU Big Basin boxes.
Re: Facebook Trains ImageNet in 1 Hour
#4tl;dr they found a clever way to spread the training across 256 GPUs by synchronising the stochastic gradient descent
Re: Facebook Trains ImageNet in 1 Hour
#5I'd always conceptualized decreasing batch size as a performance/memory optimization to deal with the fact that datasets don't all fit into memory and to reduce overall training time. You look at batch_size samples and compute the sum of the gradient of the errors to update the network weights so as to reduce the error -- shouldn't a larger batch_size inherently provide more information about the optimal direction of the update?
It seems to my naive view like it should be "nice" from an accuracy perspective to look at more samples before making an adjustment to the network weights ...?
In general, does changing the batch_size hyperparameter make a lot of difference on different problems ...? Does the right value for batch size tend to be problem specific and/or network architecture specific?
Re: Facebook Trains ImageNet in 1 Hour
#6I'd always conceptualized decreasing batch size as a performance/memory optimization to deal with the fact that datasets don't all fit into memory and to reduce overall training time. You look at batch_size samples and compute the sum of the gradient of the errors to update the network weights so as to reduce the error -- shouldn't a larger batch_size inherently provide more information about the optimal direction of…
[Warning: far from an expert here]
No, batches also help you escape local minima.
Re: Facebook Trains ImageNet in 1 Hour
#7tl;dr they found a clever way to spread the training across 256 GPUs by synchronising the stochastic gradient descent
it is a very interesting idea, considering how GD updates are inherently a serial operation. I've always wondered how they pulled it off in spark... I've also read about hogwild which allows parallel SGD on sparse datasets...
Re: Facebook Trains ImageNet in 1 Hour
#8I can't seem to find it anywhere but what is the interconnect between servers being used? NVLink is used internally for GPU to GPU communication within a single box...correct? But this sounds like it takes a cluster of 32 of their 8 GPU Big Basin boxes.
Interconnect between the servers is 50 Gbit Ethernet (see section 4 of the paper).
Re: Facebook Trains ImageNet in 1 Hour
#9How much time would it have taken using Torch instead of Caffe2? (I still don't understand which I should use...)
Re: Facebook Trains ImageNet in 1 Hour
#10Trivia: the Pieter in the paper is the one of the Redis fame.