Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook Trains ImageNet in 1 Hour

news.developer.nvidia.com

Post reply on HN
Print view View original

41–50 of 50 posts

Re: Facebook Trains ImageNet in 1 Hour

#41

Post by jamesblonde »

dgacmu wrote: post #36
nl wrote: post #31

Earlier quoted context omitted.

10 node GPU clusters are relatively common at eg Universities.

Concur. I threw 2x GTX 980s in one of our testbeds two years ago just to make it more generally useful, giving us a 20 GPU cluster. Next time we do it, I'll aim for 4x1080ti or the equivalent. It's painful to power and cool those in lower-end university machine rooms, but it's possible. 10x 4x1080ti machines is only about 15kW, which is high but not insane. It's also only about $60k USD. Also something that a pool of…

Nice. We have added support for GPUs to YARN in Hops Hadoop. And users have CPU and GPU quotas in our version of YARN. Jobs are launched as either Tensorflow on Spark or Distributed Tensorflow on YARN. If the user exceeds her quota, currently we allow them to complete. But give them a ticking off. We could also abort jobs on quota violations, but like you alluded to - after 5 days of training, aborting that job would be a world of pain for us.

Re: Facebook Trains ImageNet in 1 Hour

#42

Post by jamesblonde »

jamesblonde wrote: post #29
tanilama wrote: post #21

A practically interesting paper. Some insights: 1.Larger batches requires large learning rate. And this paper shows that learning rate can even scale linearly with the batch size, which leading to extremely large learning rate/batch sizes. 2.Larger batch causes initial learning difficult, so this paper proposes to have a warm-up period where during the initial epochs, the learning rate grows from a smaller value grad…

I disagree that this is not an architecture for others. We are a research lab, and we're building a distributed cluster. Cheap infiniband. Lots of gtx1080Tis. It's not that expensive to have a 40-GPU cluster with 10 4U servers (about 100K Euro).

You can see the potential scale-out you can get for Tensorflow here on this PR:

https://github.com/tensorflow/tensorflow/issues/2916

Re: Facebook Trains ImageNet in 1 Hour

#43

Post by Retric »

jamesblonde wrote: post #39
gm-conspiracy wrote: post #30

Earlier quoted context omitted.

What mobo/CPU are you using? How much RAM in each server? Can you provide some more detailed specs? Thanks!

CPU is not that important. A motherboard that has PCIe 3.0 and 7 PCIx16 slots is ok for at least 3 GPUs. Each GPU takes typically two slots. Then, a >1400 Watts PSU. Here is a relatively cheap box that also has 8 disk slots (for hadoop): https://exxactcorp.com/index.php/solution/solu_detail/320

Some people are using water cooling setups to get 7 GPU's on a 7 PCI slot MB. You need to cut off the DVI that would otherwise occupy another slot, but overall it's not that time consuming and reduces your networking needs.

EX: https://www.youtube.com/watch?v=9hsQmcSwGv0

Re: Facebook Trains ImageNet in 1 Hour

#44

Post by pietern »

antirez wrote: post #10

Trivia: the Pieter in the paper is the one of the Redis fame.

Hi Salvatore! :D

And there's even a tiny Redis dependency (optional though) in the code to generate these results. In particular the collective communication library needs a rendezvous phase where all nodes connect to their peers. Using Redis for this is one of the options. See: https://github.com/facebookincubator/gloo/tree/master/gloo/r...

Re: Facebook Trains ImageNet in 1 Hour

#45

Post by pietern »

jamesblonde wrote: post #16

Some observations: * for synchronous model-based distributed training to scale linearly, the time required to broadcast the model must be much larger than the time required for a worker (GPU) to process a batch * it's not strict synchronous training, as when gradients are computed at a worker, they are transmitted to all workers - so the driver doesn't have to send models to all 32 workers at the same time (8 GPUs pe…

Re: your second point, it is strictly synchronous, though since there are 8 GPUs per process (thus have 1 process per machine) the gradient reduction is done in 3 phases. First they are reduced within the process, then across processes/machines, and then broadcast within the processes.

Re: Facebook Trains ImageNet in 1 Hour

#46

Post by jamesblonde »

pietern wrote: post #45
jamesblonde wrote: post #16

Some observations: * for synchronous model-based distributed training to scale linearly, the time required to broadcast the model must be much larger than the time required for a worker (GPU) to process a batch * it's not strict synchronous training, as when gradients are computed at a worker, they are transmitted to all workers - so the driver doesn't have to send models to all 32 workers at the same time (8 GPUs pe…

Re: your second point, it is strictly synchronous, though since there are 8 GPUs per process (thus have 1 process per machine) the gradient reduction is done in 3 phases. First they are reduced within the process, then across processes/machines, and then broadcast within the processes.

I misphrased that point, agreed. It's not classic driver-driven synchronous training, as you would do in tensorflow. It's using all-reduce (not available in tensorflow yet, i think).

Re: Facebook Trains ImageNet in 1 Hour

#47

Post by randyrand »

willvarfar wrote: post #32
randyrand wrote: post #13

Earlier quoted context omitted.

this seems like the trivial, most obvious way to parallelize training across GPUs. Not, imo, clever. The important bit here is that they've shown that large mini batch sizes still can maintain accuracy if you slow the learning rate.

Lots of clever things are simply self-evident in hindsight.

Someone asked 5 days ago on HN how to parallelize gpu learning. I had never thought about the problem before, but still came up with and gave this as the most obvious way. Took me maybe 20 seconds to think of.

https://news.ycombinator.com/item?id=14510146

So the fact that someone with little experience can come up with this 'clever' technique, means either I'm really clever or it's not that clever. I'll go with the latter.

Re: Facebook Trains ImageNet in 1 Hour

#48

Post by randyrand »

opportune wrote: post #35
randyrand wrote: post #13

Earlier quoted context omitted.

this seems like the trivial, most obvious way to parallelize training across GPUs. Not, imo, clever. The important bit here is that they've shown that large mini batch sizes still can maintain accuracy if you slow the learning rate.

Just because it's simple to explain at a high level doesn't make it trivial. Plenty of theoretically trivial solutions to problems are absolute pains to implement. I mean there are entire companies that at their core solve relatively "trivial" problems but employ huge numbers of engineers. Just because the core concept is simple to explain doesn't mean it's easy.

Someone asked the other day how to parallelize gpu learning. I had never thought about the problem before, but still came up with and gave this as the most obvious way.

https://news.ycombinator.com/item?id=14510146

Re: Facebook Trains ImageNet in 1 Hour

#50

Post by antirez »

pietern wrote: post #44
antirez wrote: post #10

Trivia: the Pieter in the paper is the one of the Redis fame.

Hi Salvatore! :D And there's even a tiny Redis dependency (optional though) in the code to generate these results. In particular the collective communication library needs a rendezvous phase where all nodes connect to their peers. Using Redis for this is one of the options. See: https://github.com/facebookincubator/gloo/tree/master/gloo/r...

Hey Pieter! Wow cool :-) Thanks for the info. See you soon!
Post reply on HN