Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TensorFlow Benchmarks

github.com

Post reply on HN
Print view View original

31–40 of 57 posts

Re: TensorFlow Benchmarks

#31

Post by dave_sullivan »

kastnerkyle wrote: post #28
dave_sullivan wrote: post #24

Earlier quoted context omitted.

I'm going to say something unpopular, but horizontally-scaled deep learning is overkill for most applications. Can anyone here present a use case where they have personally needed horizontal scaling because a Titan X couldn't fit what they were trying to do? It's the thing I hear talked about the most and used the least. The biggest misunderstanding I've heard is "I have petabytes of data so I need multi-GPU", but NN…

For big LSTMs and long-ish sequences, the intermediate gradients can take up a huge amount of memory - often more than the model parameters themselves. In my experience it is mostly big LSTMs that need the 12GB+ GPUs. You can reduce the batch size to help this a bit, or train using trucated BPTT but RNN training is already a slow, sequential business. Of course, there are no clear wins (generally, losses) in computat…

All true things. But at the heart of the issue:

    No clear wins in horizontal scaling

    Reduce batch size

    Use truncated back prop

    *search for better hyper params*
I usually do 2-4. After those, have you really seen scaling result in significant accuracy gains? And what percent of the time is that necessary? Genuinely interested--and you guys rock btw!

Re: TensorFlow Benchmarks

#32

Post by mtw »

discardorama wrote: post #22
argonaut wrote: post #20

Earlier quoted context omitted.

I think I'll give the benefit of the doubt to, you know, the pioneer of deep learning, inventor of convolutional neural networks, and (co)-inventor of the backpropagation algorithm.

Just because you invent an algorithm, it doesn't mean you know how to implement it in the most efficient way possible. I think the key to TensorFlow is not how fast it runs on 1 machine; but how fast it runs on 10,000. Consider map-reduce (Hadoop). Sure, you can sort 1GB data on a single machine 10x faster (using /usr/bin/sort) than using Hadoop on that machine; but make the data 1TB and add 1000 machines, now lets s…

Scenarios of machine learning on 1 machine is way more common than machine learning on 10,000 machines! I doubt even Google would have 10,000 for google translate

It's more realistic to discuss librairies for 1 machine / GPU

Re: TensorFlow Benchmarks

#33

Post by kastnerkyle »

dave_sullivan wrote: post #31
kastnerkyle wrote: post #28

Earlier quoted context omitted.

For big LSTMs and long-ish sequences, the intermediate gradients can take up a huge amount of memory - often more than the model parameters themselves. In my experience it is mostly big LSTMs that need the 12GB+ GPUs. You can reduce the batch size to help this a bit, or train using trucated BPTT but RNN training is already a slow, sequential business. Of course, there are no clear wins (generally, losses) in computat…

All true things. But at the heart of the issue: No clear wins in horizontal scaling Reduce batch size Use truncated back prop *search for better hyper params* I usually do 2-4. After those, have you really seen scaling result in significant accuracy gains? And what percent of the time is that necessary? Genuinely interested--and you guys rock btw!

One case where I see 1 as being necessary is the softmax size / vocabulary boost in the seq2seq paper (8 GPUs IIRc, and 4 were dedicated to a softmax!). There are other ways to handle this (hierarchical softmax, sampled softmax, skip-thoughts trick of using word2vec vocabulary) but every time someone figures out how to have a larger vocabulary, neural machine translation results seem to improve. In general, 1 is a last resort for me - but maybe this is due to current tooling and availability of hardware as much as anything?

Re: TensorFlow Benchmarks

#34

Post by mtw »

mark_l_watson wrote: post #10

From Google's perspective it is probably more about how TensorFlow scales out horizontally. If a researcher fires off a Borg run (or whatever they use now) and the job takes a few thousand CPUs, no problem, at least for research. They must have better optimization a for running in production, such as in place operations.

The rest of us don't have "a few thousand CPUs" though. Seen this way, TensorFlow is largely inaccessible and makes more sense to focus on existing librairies optimized for single cpus

Re: TensorFlow Benchmarks

#35

Post by smhx »

kastnerkyle wrote: post #30
donthateyo wrote: post #13

Until now, I've seen two responses to Google's TensorFlow from Facebook employees. Yann Le Cunn seemed to really challenge Jeff Dean about TensorFlow's scalability [1] and this benchmark puts TensorFlow down there in all the measures it tested for. I can't ignore the possibility that this criticism of TensorFlow from Facebook employees (while factually correct and constructive) might be driven by some competition and…

This benchmark basically shows that releasing TensorFlow with cudnn v2 backend support hurts - v2 is quite a bit slower than v3 (current) and v4 (upcoming). TF has announced that they will update to v4 support, which should help quite a bit - but when many hobbyists and researchers are developing on one or two GPUs performance on that scale is more important (for them) than infinite scalability. It is not surprising…

I've done an apples to apples, TensorFlow + CuDNN R2 vs Torch + CuDNN R2

Re: TensorFlow Benchmarks

#36

Post by kajecounterhack »

dave_sullivan wrote: post #24
mark_l_watson wrote: post #10

From Google's perspective it is probably more about how TensorFlow scales out horizontally. If a researcher fires off a Borg run (or whatever they use now) and the job takes a few thousand CPUs, no problem, at least for research. They must have better optimization a for running in production, such as in place operations.

I'm going to say something unpopular, but horizontally-scaled deep learning is overkill for most applications. Can anyone here present a use case where they have personally needed horizontal scaling because a Titan X couldn't fit what they were trying to do? It's the thing I hear talked about the most and used the least. The biggest misunderstanding I've heard is "I have petabytes of data so I need multi-GPU", but NN…

> horizontally-scaled deep learning is overkill for most applications

It's difficult to say what "most applications are" but many algos currently seem bottlenecked by how fast you can load data into your GFX card (hint: it's slow). So what you're saying about batching is correct, but it does matter.

Re: TensorFlow Benchmarks

#37

Post by smhx »

donthateyo wrote: post #13

Until now, I've seen two responses to Google's TensorFlow from Facebook employees. Yann Le Cunn seemed to really challenge Jeff Dean about TensorFlow's scalability [1] and this benchmark puts TensorFlow down there in all the measures it tested for. I can't ignore the possibility that this criticism of TensorFlow from Facebook employees (while factually correct and constructive) might be driven by some competition and…

I'm the one who runs the benchmarks. It's sad that you put such a twist to the whole thing.

I've been running convnet-benchmarks forever now, and I've been running them independently on separate personal hardware. I do this as a hobby.

I've done an apples-to-apples comparison, and my benchmark review only puts the facts forward, I dont attack them.

If you read my other social media comments, I've been pretty positive about TensorFlow, and I've even put in some groundwork to write Torch bindings for it.

Please stop spewing nonsense interpolated from like 2 super-weak data points.

Re: TensorFlow Benchmarks

#38

Post by johanneskanybal »

mtgx wrote: post #3

It's almost like Google wanted everyone to use slow obsolete software and keep the really good stuff for itself, while still making it look like they're doing a great thing for the community.

Like a factor of 4 matters much if your framework makes it easier to scale out horizontally. Also, I'm sure it will get optimized over time.

Re: TensorFlow Benchmarks

#39

Post by johanneskanybal »

mtw wrote: post #34
mark_l_watson wrote: post #10

From Google's perspective it is probably more about how TensorFlow scales out horizontally. If a researcher fires off a Borg run (or whatever they use now) and the job takes a few thousand CPUs, no problem, at least for research. They must have better optimization a for running in production, such as in place operations.

The rest of us don't have "a few thousand CPUs" though. Seen this way, TensorFlow is largely inaccessible and makes more sense to focus on existing librairies optimized for single cpus

I don't think you understand what this library is for. You can pick up a graphic card for $100 and voila now you got 1000's of cores. To your defense the comment you answered seems to have mixed a few random things together too.

Re: TensorFlow Benchmarks

#40

Post by mtw »

johanneskanybal wrote: post #39
mtw wrote: post #34

Earlier quoted context omitted.

The rest of us don't have "a few thousand CPUs" though. Seen this way, TensorFlow is largely inaccessible and makes more sense to focus on existing librairies optimized for single cpus

I don't think you understand what this library is for. You can pick up a graphic card for $100 and voila now you got 1000's of cores. To your defense the comment you answered seems to have mixed a few random things together too.

Sorry, a CUDA core is not a CPU. See https://en.wikipedia.org/wiki/CUDA#/media/File:CUDA_processi...
Post reply on HN