Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TensorFlow Benchmarks

github.com

Post reply on HN
Print view View original

51–57 of 57 posts

Re: TensorFlow Benchmarks

#51

Post by kastnerkyle »

rryan wrote: post #47
kastnerkyle wrote: post #29

Earlier quoted context omitted.

Yann LeCun has been working on hardware implementations of convnets almost since the beginning. LeNet 5 (the check reader of ATT, circa early 90s) needed a dedicated and specialized hardware implementation IIRC. It could be kneejerk, but he definitely has the background to make a fair assessment - and this benchmark seems to back his claims.

> and this benchmark seems to back his claims. These benchmarks evaluate single-node performance. LeCun's remarks were concerning distributed training (specifically that bandwidth between machines is a limiting factor to scalability) -- which we can't test yet since the current version of TF is single-node only. Dean's response in the video "it depends on your [computer] network" is an interesting response :).

Both Google and FB (from what I understand) have a ton of tricks to help this, but I expect Yann was speaking generally even with all these tricks, and he is right from what I have seen. The old paper on DistBelief talks about topping out at ~80 machines due to network overhead - it would be great if they talk more about distributed TF in an upcoming paper.

If TF really has a way to make general, networked, distributed training efficient (more than 1 bit weight updates, low precision weights and all the other crazy tricks which already exist) - that is truly remarkable and they rightly deserve huge kudos.

If they are faster in distributed training only on Google machines or with Google's network architecture, that isn't really a useful datapoint for the general public.

We could wire everything with 10GB/s NiCs, change to jumbo frames, pull all these other bandwidth reducing tricks, trick out the Linux kernel, etc. and then your network probably won't matter - but that isn't really general or cheap. The key will be what is the minimum effort necessary to avoid network bottlenecking, and does TF improve that minimum level over existing solutions?

Re: TensorFlow Benchmarks

#52

Post by kastnerkyle »

varelse wrote: post #49
kastnerkyle wrote: post #30

Earlier quoted context omitted.

This benchmark basically shows that releasing TensorFlow with cudnn v2 backend support hurts - v2 is quite a bit slower than v3 (current) and v4 (upcoming). TF has announced that they will update to v4 support, which should help quite a bit - but when many hobbyists and researchers are developing on one or two GPUs performance on that scale is more important (for them) than infinite scalability. It is not surprising…

You might be seeing the final gasps of Google's longstanding and now-reversed anti-GPU stance in TF's initial GPU performance. NVIDIA support alone will make sure TF knocks it out of the park down the road. IMO it's crazy to consider the first release the final say on TF's GPU performance. Caffe, Torch, and Theano have a huge head start (and lots of pre-existing technical support from NVIDIA). The biggest limitation…

I don't think anyone is claiming TF won't get much, much faster (probably very soon). But claiming that right now - today - TF kills existing toolkits like Torch, Caffe, and Theano (which I have seen here and elsewhere - though not claiming you are in this camp) is a bit premature.

Even with v4 support which is coming soon, the general setup that users have is one to two GPUs in one machine. This means adding the right inplace operations can have a huge impact on performance, and is probably a usecase that Google has not focused on given their internal infrastructure. I am sure they will normalize to "at or slightly above" the performance of other toolkits, but the question is when?

This benchmark doesn't have anything to do with multi-GPU as far as I am aware - these are single machine, single GPU results. I would wager 90% of the deep learning hobbyist and research communities run in this setting, so benchmarking this is really important.

For people with huge amounts of networked and distributed resources - distributed support will be very amazing!

Re: TensorFlow Benchmarks

#53

Post by varelse »

dave_sullivan wrote: post #42
kajecounterhack wrote: post #36

Earlier quoted context omitted.

> horizontally-scaled deep learning is overkill for most applications It's difficult to say what "most applications are" but many algos currently seem bottlenecked by how fast you can load data into your GFX card (hint: it's slow). So what you're saying about batching is correct, but it does matter.

> many algos currently seem bottlenecked by how fast you can load data into your GFX card All of them are, but that's a PCI-e issue and horizontal scaling doesn't fix that (unless using nvlink or similar afaik, but then you face the fact that current horizontal scaling schemes aren't very effective at increasing model accuracy anyway) > It's difficult to say what "most applications are" Nevermind "most applications"-…

>do you use multi-GPU for your DL work? If so, how often?

I'm using it right now, to great effect. I can't really say what for, but methinks I'll be using it every hour of every day for the foreseeable future.

Re: TensorFlow Benchmarks

#54

Post by varelse »

kastnerkyle wrote: post #52
varelse wrote: post #49

Earlier quoted context omitted.

You might be seeing the final gasps of Google's longstanding and now-reversed anti-GPU stance in TF's initial GPU performance. NVIDIA support alone will make sure TF knocks it out of the park down the road. IMO it's crazy to consider the first release the final say on TF's GPU performance. Caffe, Torch, and Theano have a huge head start (and lots of pre-existing technical support from NVIDIA). The biggest limitation…

I don't think anyone is claiming TF won't get much, much faster (probably very soon). But claiming that right now - today - TF kills existing toolkits like Torch, Caffe, and Theano (which I have seen here and elsewhere - though not claiming you are in this camp) is a bit premature. Even with v4 support which is coming soon, the general setup that users have is one to two GPUs in one machine. This means adding the rig…

Inplace is a nice late phase optimization, but the root problem here appears to be that Google's convolution kernels are crap compared to those in cuDNN3 and Neon (I asked Scott Gray about this directly and I trust his wisdom).

No surprises there whatsoever. The tensorflow engine is the most gold-plated POS I've seen in a long time. if it were running on 1000+ servers, I'd get the level of overengineering they've applied here. But single server pthreads? WTF?

Also parameter server is dumb unless you sweat the implementation of the gather/reduction ops, but I digress.

Re: TensorFlow Benchmarks

#55

Post by yjxiong »

freerobby wrote: post #25
dave_sullivan wrote: post #24

Earlier quoted context omitted.

I'm going to say something unpopular, but horizontally-scaled deep learning is overkill for most applications. Can anyone here present a use case where they have personally needed horizontal scaling because a Titan X couldn't fit what they were trying to do? It's the thing I hear talked about the most and used the least. The biggest misunderstanding I've heard is "I have petabytes of data so I need multi-GPU", but NN…

Would image training for video recognition be a good example of this?

Our experiments might support this. With 4 or 8 titan X, training time is shortened from more than one week to one or two days. code available at https://github.com/yjxiong/caffe

Re: TensorFlow Benchmarks

#56

Post by dave_sullivan »

yjxiong wrote: post #55
freerobby wrote: post #25

Earlier quoted context omitted.

Would image training for video recognition be a good example of this?

Our experiments might support this. With 4 or 8 titan X, training time is shortened from more than one week to one or two days. code available at https://github.com/yjxiong/caffe

Note: data parallel vs model parallel.

Data parallel is easy to implement and leads to linear training time speed ups, as per 1 week to 2 days with 4x hardware. But not bigger models.

Model parallelism leads to bigger models and is what I've been referring to. It is overkill and doesn't work well anyway. Frameworks should not be expected to support it because there are more interesting topics in deep learning they could support instead.

Better research/methods will come out at some point, at which point this calculus will change, but not yet! Today it is the very definition of premature optimization in nearly all cases.

Re: TensorFlow Benchmarks

#57

Post by blazespin »

varelse wrote: post #54
kastnerkyle wrote: post #52

Earlier quoted context omitted.

I don't think anyone is claiming TF won't get much, much faster (probably very soon). But claiming that right now - today - TF kills existing toolkits like Torch, Caffe, and Theano (which I have seen here and elsewhere - though not claiming you are in this camp) is a bit premature. Even with v4 support which is coming soon, the general setup that users have is one to two GPUs in one machine. This means adding the rig…

Inplace is a nice late phase optimization, but the root problem here appears to be that Google's convolution kernels are crap compared to those in cuDNN3 and Neon (I asked Scott Gray about this directly and I trust his wisdom). No surprises there whatsoever. The tensorflow engine is the most gold-plated POS I've seen in a long time. if it were running on 1000+ servers, I'd get the level of overengineering they've app…

Do we know if this is a just a reference implementation or what Google uses in production? My guess the numbers will come down pretty quick. From what I can see from mxnet versus theano versus torch is that GPUs are really the final determinant of speed and not the framework.
Post reply on HN