Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TensorFlow Benchmarks

github.com

Post reply on HN
Print view View original

41–50 of 57 posts

Re: TensorFlow Benchmarks

#41

Post by varelse »

dave_sullivan wrote: post #24
mark_l_watson wrote: post #10

From Google's perspective it is probably more about how TensorFlow scales out horizontally. If a researcher fires off a Borg run (or whatever they use now) and the job takes a few thousand CPUs, no problem, at least for research. They must have better optimization a for running in production, such as in place operations.

I'm going to say something unpopular, but horizontally-scaled deep learning is overkill for most applications. Can anyone here present a use case where they have personally needed horizontal scaling because a Titan X couldn't fit what they were trying to do? It's the thing I hear talked about the most and used the least. The biggest misunderstanding I've heard is "I have petabytes of data so I need multi-GPU", but NN…

I suspect this is a just-so-tale because the capability is so new and the GPU programming skill required to implement it efficiently is so rare. TLDR: just say no to parameter server (most of the time).

But IMO the real obstacle to horizontal scaling is the communication between servers not the usefulness of doing so. Within a server, one can pack 8 TitanX/M40 GPUs with high speed 13 GB/s P2P communication between them (and up to 16 GPUs in various unproven science project servers). That's ~50 TFLOPs and 96 GB in a box. That rocks. Just ask the guys at mindori (if they ever ship that is).

http://www.mindori.com

But between servers lies a sippy straw of 100 Gb/s Infiniband at best or worst-case, ~1 Gb/s on AWS GPU servers with freaky nearest neighbor weather. If you can't make efficient use of 8 GPUs in a single box, I agree, don't bother breaking out to the next server.

That said, frameworks like mxnet have opened the floodgates to experimenting with larger models and more distributed training algorithms. Time will tell if this pans out. But 2 years ago, Andrew Ng's group showed 12 GTX 680s distributed across 3 servers kicking the crap out of Google Brain. I expect more of this, not less, in the near future.

Re: TensorFlow Benchmarks

#42

Post by dave_sullivan »

kajecounterhack wrote: post #36
dave_sullivan wrote: post #24

Earlier quoted context omitted.

I'm going to say something unpopular, but horizontally-scaled deep learning is overkill for most applications. Can anyone here present a use case where they have personally needed horizontal scaling because a Titan X couldn't fit what they were trying to do? It's the thing I hear talked about the most and used the least. The biggest misunderstanding I've heard is "I have petabytes of data so I need multi-GPU", but NN…

> horizontally-scaled deep learning is overkill for most applications It's difficult to say what "most applications are" but many algos currently seem bottlenecked by how fast you can load data into your GFX card (hint: it's slow). So what you're saying about batching is correct, but it does matter.

> many algos currently seem bottlenecked by how fast you can load data into your GFX card

All of them are, but that's a PCI-e issue and horizontal scaling doesn't fix that (unless using nvlink or similar afaik, but then you face the fact that current horizontal scaling schemes aren't very effective at increasing model accuracy anyway)

> It's difficult to say what "most applications are"

Nevermind "most applications"--so far, all I've heard is one, that being the absolute bleeding edge of RNN research, assuming you're using a huge softmax instead of an alternative.

My point remains: Multi-GPU is way down the list when it comes to features a DL framework should have. Because very very few people need it.

kajecounterhack, do you use multi-GPU for your DL work? If so, how often?

Re: TensorFlow Benchmarks

#43

Post by varelse »

mtw wrote: post #40
johanneskanybal wrote: post #39

Earlier quoted context omitted.

I don't think you understand what this library is for. You can pick up a graphic card for $100 and voila now you got 1000's of cores. To your defense the comment you answered seems to have mixed a few random things together too.

Sorry, a CUDA core is not a CPU. See https://en.wikipedia.org/wiki/CUDA#/media/File:CUDA_processi...

No but an SMX is the GPU equivalent of a CPU core... And there are 24 of them in a GM200...

Re: TensorFlow Benchmarks

#44

Post by varelse »

mtw wrote: post #34
mark_l_watson wrote: post #10

From Google's perspective it is probably more about how TensorFlow scales out horizontally. If a researcher fires off a Borg run (or whatever they use now) and the job takes a few thousand CPUs, no problem, at least for research. They must have better optimization a for running in production, such as in place operations.

The rest of us don't have "a few thousand CPUs" though. Seen this way, TensorFlow is largely inaccessible and makes more sense to focus on existing librairies optimized for single cpus

TensorFlow is currently a single-server solution pthreads app.

But assuming you're serious and not dabbling, you can build a ~27 TFLOP server for http://exxactcorp.com/index.php/solution/solu_detail/225

Or if you just hate the thought of money burning a hole in your pocket, pay NVIDIA >$15K for the same thing with a pretty NVIDIA logo:

https://developer.nvidia.com/devbox

If you're dabbling, just go buy a $1000 TitanX and you're instantly a minor deep learning superpower.

Re: TensorFlow Benchmarks

#45

Post by moultano »

dave_sullivan wrote: post #24
mark_l_watson wrote: post #10

From Google's perspective it is probably more about how TensorFlow scales out horizontally. If a researcher fires off a Borg run (or whatever they use now) and the job takes a few thousand CPUs, no problem, at least for research. They must have better optimization a for running in production, such as in place operations.

I'm going to say something unpopular, but horizontally-scaled deep learning is overkill for most applications. Can anyone here present a use case where they have personally needed horizontal scaling because a Titan X couldn't fit what they were trying to do? It's the thing I hear talked about the most and used the least. The biggest misunderstanding I've heard is "I have petabytes of data so I need multi-GPU", but NN…

>Can anyone here present a use case where they have personally needed horizontal scaling

The embeddings for text models with a real world vocabulary can get very very large, especially if multiple languages are involved.

Re: TensorFlow Benchmarks

#46

Post by emcq »

Is this surprising? I think it's expected, when most published numbers from Google suggest that this is only 2x over DistBelief, which Adam Coates improved upon 6x (http://www.cs.stanford.edu/~acoates/papers/CoatesHuvalWangWu...) in 2013. Coming from a HPC background, DistBelief (and sometimes MR in general) has always felt like a square being forced into a round hole. Sure it "scales" but it's not how you would build a system from scratch to meet the same computational needs. Moving data around is much slower than doing extra math.

DNNs provide some interesting opportunities for domain specific optimization with lossy math because empirically it seems that the solutions dont require significant precision. Something like a general Tensor library holds you back from using some of the tricks like synchronization-free hogwild, where it wouldn't be appropriate in other scenarios where more exact computation is necessary.

One interesting step towards this is their implementation of lossy compression with a custom 16-bit half float representation. However a lot of people are dialing down the precision even further, including one of the authors of TensorFlow (http://petewarden.com/2015/05/23/why-are-eight-bits-enough-f...).

Scott Gray, one of the principle engineers at Nervana Systems, has a very wise reply in the github thread - they aren't doing much to optimize for memory locality, and wont have amazing speedups until that happens.

Re: TensorFlow Benchmarks

#47

Post by rryan »

kastnerkyle wrote: post #29
hueving wrote: post #27

Earlier quoted context omitted.

>I'm just pointing out how ridiculous it is to cast aspersions on the judgment of one of the world's leading experts on deep learning. A position of authority doesn't mean the person is immune to jealousy. Someone in a high status position is going to be much more likely to have a knee-jerk defensive reaction to news that threatens their image.

Yann LeCun has been working on hardware implementations of convnets almost since the beginning. LeNet 5 (the check reader of ATT, circa early 90s) needed a dedicated and specialized hardware implementation IIRC. It could be kneejerk, but he definitely has the background to make a fair assessment - and this benchmark seems to back his claims.

> and this benchmark seems to back his claims.

These benchmarks evaluate single-node performance. LeCun's remarks were concerning distributed training (specifically that bandwidth between machines is a limiting factor to scalability) -- which we can't test yet since the current version of TF is single-node only.

Dean's response in the video "it depends on your [computer] network" is an interesting response :).

Re: TensorFlow Benchmarks

#48

Post by rspeer »

discardorama wrote: post #22
argonaut wrote: post #20

Earlier quoted context omitted.

I think I'll give the benefit of the doubt to, you know, the pioneer of deep learning, inventor of convolutional neural networks, and (co)-inventor of the backpropagation algorithm.

Just because you invent an algorithm, it doesn't mean you know how to implement it in the most efficient way possible. I think the key to TensorFlow is not how fast it runs on 1 machine; but how fast it runs on 10,000. Consider map-reduce (Hadoop). Sure, you can sort 1GB data on a single machine 10x faster (using /usr/bin/sort) than using Hadoop on that machine; but make the data 1TB and add 1000 machines, now lets s…

> make the data 1TB and add 1000 machines, now lets see how fast you can sort with /usr/bin/sort!

Still faster than Hadoop, because sorting the data will be faster than sending it over a network. What should I do with the other 999 machines?

(I know what you actually mean, by the way. Make it 100 TB and say that the data is already spread out across 1000 machines.)

Re: TensorFlow Benchmarks

#49

Post by varelse »

kastnerkyle wrote: post #30
donthateyo wrote: post #13

Until now, I've seen two responses to Google's TensorFlow from Facebook employees. Yann Le Cunn seemed to really challenge Jeff Dean about TensorFlow's scalability [1] and this benchmark puts TensorFlow down there in all the measures it tested for. I can't ignore the possibility that this criticism of TensorFlow from Facebook employees (while factually correct and constructive) might be driven by some competition and…

This benchmark basically shows that releasing TensorFlow with cudnn v2 backend support hurts - v2 is quite a bit slower than v3 (current) and v4 (upcoming). TF has announced that they will update to v4 support, which should help quite a bit - but when many hobbyists and researchers are developing on one or two GPUs performance on that scale is more important (for them) than infinite scalability. It is not surprising…

You might be seeing the final gasps of Google's longstanding and now-reversed anti-GPU stance in TF's initial GPU performance.

NVIDIA support alone will make sure TF knocks it out of the park down the road. IMO it's crazy to consider the first release the final say on TF's GPU performance. Caffe, Torch, and Theano have a huge head start (and lots of pre-existing technical support from NVIDIA).

The biggest limitation I see right now is that their multi-GPU algorithms are really simple and inefficient. That will change I'm sure now that they're getting benchmarked against everyone else.

Re: TensorFlow Benchmarks

#50

Post by kastnerkyle »

smhx wrote: post #35
kastnerkyle wrote: post #30

Earlier quoted context omitted.

This benchmark basically shows that releasing TensorFlow with cudnn v2 backend support hurts - v2 is quite a bit slower than v3 (current) and v4 (upcoming). TF has announced that they will update to v4 support, which should help quite a bit - but when many hobbyists and researchers are developing on one or two GPUs performance on that scale is more important (for them) than infinite scalability. It is not surprising…

I've done an apples to apples, TensorFlow + CuDNN R2 vs Torch + CuDNN R2

My point is more that even if they were on the same footing from benchmark timings, v2 is still far behind what is supported in Torch, Caffe, and Theano right now (v3 in all IIRC). Your comparison is very fair, and it is good insight!
Post reply on HN