Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook Trains ImageNet in 1 Hour

news.developer.nvidia.com

Post reply on HN
Print view View original

21–30 of 50 posts

Re: Facebook Trains ImageNet in 1 Hour

#21

Post by tanilama »

A practically interesting paper. Some insights:

1.Larger batches requires large learning rate. And this paper shows that learning rate can even scale linearly with the batch size, which leading to extremely large learning rate/batch sizes.

2.Larger batch causes initial learning difficult, so this paper proposes to have a warm-up period where during the initial epochs, the learning rate grows from a smaller value gradually to a larger one.

But if you are not Google/Facebook/Amazon/Microsoft, the experiment setting is unrealistic to you. Best AWS instances didn't come with the 50GBits network. For now, for the others, we would still stick to at most 8 GPUs on a single node, even your soul screams distributed :/

Re: Facebook Trains ImageNet in 1 Hour

#23

Post by IanCal »

breatheoften wrote: post #5

I'd always conceptualized decreasing batch size as a performance/memory optimization to deal with the fact that datasets don't all fit into memory and to reduce overall training time. You look at batch_size samples and compute the sum of the gradient of the errors to update the network weights so as to reduce the error -- shouldn't a larger batch_size inherently provide more information about the optimal direction of…

> shouldn't a larger batch_size inherently provide more information about the optimal direction of the update?

Not necessarily, since a batch gradient output (as I understand it, and at least used to code it) all gets averaged together.

Consider standing in a valley with two equal hills either side of you. If you were to try one direction and see that climbing that way helps, you'd take a step that way. Then the next step would keep taking you up that hill.

Now, if you batched together two direction tests, what would happen? You'd average together your left and right and end up with moving nowhere. Having both at the same time doesn't give you better information about how you move if you only see the result after averaging.

This interestingly maps to something we see in humans, though I'm struggling to find a decent paper on it (from the PRISM lab in Birmingham, UK if anyone else has any luck, think the person doing the research might have been called Chris). Simple adaptation tasks, in this case learning to control a joystick that has a clockwise/anticlockwise force applied to it, don't work well if you try and learn both one thing and the opposite straight away. However, sleeping in-between learning each left you able to do both well. Perhaps this was early results though.

Batch tradeoffs:

https://stats.stackexchange.com/questions/164876/tradeoff-ba...

https://arxiv.org/abs/1609.04836

Re: Facebook Trains ImageNet in 1 Hour

#24

Post by jononor »

IanCal wrote: post #23
breatheoften wrote: post #5

I'd always conceptualized decreasing batch size as a performance/memory optimization to deal with the fact that datasets don't all fit into memory and to reduce overall training time. You look at batch_size samples and compute the sum of the gradient of the errors to update the network weights so as to reduce the error -- shouldn't a larger batch_size inherently provide more information about the optimal direction of…

> shouldn't a larger batch_size inherently provide more information about the optimal direction of the update? Not necessarily, since a batch gradient output (as I understand it, and at least used to code it) all gets averaged together. Consider standing in a valley with two equal hills either side of you. If you were to try one direction and see that climbing that way helps, you'd take a step that way. Then the next…

If this is the case, could one get improved learning by mixing large and small batches?

Re: Facebook Trains ImageNet in 1 Hour

#26

Post by spuz »

rlv-dan wrote: post #25

Does anyone know if there are any "consumer grade" image training kit out there? I'm thinking a software that you can train on your own images to put into categories.

Yes, you can use Tensorflow and Google's "inception" image recognition model to do this. The model by default is trained on the Imagenet database of images/categories, but Tensorflow allows you to retrain the last layer of the model on your own images to produce your own categorisation. Since you are only retraining the last layer of the model, you can easily do it within about 20 minutes on a laptop. See the tutorial here: https://www.tensorflow.org/tutorials/image_retraining

Re: Facebook Trains ImageNet in 1 Hour

#27

Post by Eridrus »

horsecaptin wrote: post #11

Looks like Nvidia is doing a proper content marketing push to compete against AMD.

Interestingly enough I think the people who gain the most from this paper are cloud providers. Very few orgs will buy 256 GPUs, but with linear scaling, renting them makes a lot of sense.

Re: Facebook Trains ImageNet in 1 Hour

#28

Post by cs702 »

tanilama wrote: post #21

A practically interesting paper. Some insights: 1.Larger batches requires large learning rate. And this paper shows that learning rate can even scale linearly with the batch size, which leading to extremely large learning rate/batch sizes. 2.Larger batch causes initial learning difficult, so this paper proposes to have a warm-up period where during the initial epochs, the learning rate grows from a smaller value grad…

The approach could prove useful in a single machine too -- for example, in cases where increasing batch size might improve the efficiency of shuffling data back and forth from main memory to GPU memory. This provides an easy recipe for trying larger batch sizes to increase GPU usage.

Re: Facebook Trains ImageNet in 1 Hour

#29

Post by jamesblonde »

tanilama wrote: post #21

A practically interesting paper. Some insights: 1.Larger batches requires large learning rate. And this paper shows that learning rate can even scale linearly with the batch size, which leading to extremely large learning rate/batch sizes. 2.Larger batch causes initial learning difficult, so this paper proposes to have a warm-up period where during the initial epochs, the learning rate grows from a smaller value grad…

I disagree that this is not an architecture for others. We are a research lab, and we're building a distributed cluster. Cheap infiniband. Lots of gtx1080Tis. It's not that expensive to have a 40-GPU cluster with 10 4U servers (about 100K Euro).

Re: Facebook Trains ImageNet in 1 Hour

#30

Post by gm-conspiracy »

jamesblonde wrote: post #29
tanilama wrote: post #21

A practically interesting paper. Some insights: 1.Larger batches requires large learning rate. And this paper shows that learning rate can even scale linearly with the batch size, which leading to extremely large learning rate/batch sizes. 2.Larger batch causes initial learning difficult, so this paper proposes to have a warm-up period where during the initial epochs, the learning rate grows from a smaller value grad…

I disagree that this is not an architecture for others. We are a research lab, and we're building a distributed cluster. Cheap infiniband. Lots of gtx1080Tis. It's not that expensive to have a 40-GPU cluster with 10 4U servers (about 100K Euro).

What mobo/CPU are you using? How much RAM in each server?

Can you provide some more detailed specs?

Thanks!

Post reply on HN