Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Looming Battle Over AI Chips

barrons.com

Post reply on HN
Print view View original

61–70 of 89 posts

Re: The Looming Battle Over AI Chips

#61

Post by joe_the_user »

Symmetry wrote: post #58
joe_the_user wrote: post #20

Earlier quoted context omitted.

Remember that even though NVidia calls each execution port a "Core" it can only execute one instruction across all of them at a time. There are clever ways around this limitation, see links in my post this thread. https://news.ycombinator.com/item?id=16892107

Those are some really clever ways to make sure that all the threads in your program are executing the same instruction, but it doesn't get around the problem. Thanks for linking that video, though.

The key of the Dietz system (MOG) is that the native code that the GPU runs is a bytecode interpreter. Bytecode "instruction pointer" together with other data is just data in registers and memory that's interpreted by the native code interpreter. So for each thread, the instruction pointer can point at a different command - the interpreter runs the same instructions but the results are different. So effectively you are simulating a general purpose CPU running a different instruction on each thread. There are further tricks required to make this efficient, of course. But you are effectively running a different general purpose instruction per thread (actually runs MIPS assembler I recall).

Re: The Looming Battle Over AI Chips

#62

Post by joshuamorton »

oneshot908 wrote: post #56
joshuamorton wrote: post #51

Earlier quoted context omitted.

I'm a little confused here, are you saying that ML ASICs can't beat compute per $ of GPUs? That seems, on its face, to be a ridiculous assertion, so I'm confused where I'm misunderstanding you.

No, they can't when you can get tens of TFLOPS per GPU for Now when a Tesla GPU costs $9-10K instead of In contrast, if you can right-size the transistor count for expensive and very specific workloads across 100K+ machines like companies with big datacenters and big data sets can do, I see an opportunity for all of the BigCos to build custom ASICs for their siloed data sets and graphs. That's what GOOG is doing and…

Yes but over the lifetime of a GPU, you'll spend more on power draw than the physical hardware. That's where the savings come from, or at least that's what I've been told.

A V100 costs ~300/yr in electricity. If you are buying at the scale of 100k units, but can price per operation, even by just 10% (for example, by dropping features you don't care about), that's a million dollars of electricity over the lifetime of your hardware.

Re: The Looming Battle Over AI Chips

#63

Post by emcq »

joshuamorton wrote: post #51
oneshot908 wrote: post #14

If you're FB, GOOG, AAPL, AMZN, BIDU, etc, this strategy makes sense because much like they have siloed data, they also have siloed computation graphs for which they can lovingly design artisan transistors to make the perfect craft ASIC. There's big money in this. Or you can be like BIDU, buy 100K consumer GPUs, and put them in your datacenter. In response, Jensen altered the CUDA 9.1 licensing agreement and the EULA…

I'm a little confused here, are you saying that ML ASICs can't beat compute per $ of GPUs? That seems, on its face, to be a ridiculous assertion, so I'm confused where I'm misunderstanding you.

At a high level there is a design tradeoff where you put your transistors for a given chip. For a dense linear algebra/tensor processor, it basically comes down to using your transistors for memory or compute.

GPUs (and DSPs) historically are way on the compute side. You get kilobytes of on chip memory, and really fat parallel buses to off chip RAM.

On the other end, you have some chips that put more memory near the compute. It means you have less compute, but way better power efficiency. Each hop from a register to cache to off chip to between boards to between nodes is roughly 2-10x power hit, so you get orders of magnitudes here.

In terms of training, it's really hard to fit your training data on chip. In which case you end up with an architecture very similar to a GPU or DSP/TPU. NVIDIA is no slouch here. A couple years ago the big trick was reducing precision - you dont need single or double floats during training so an architecture more specialized at fp16 or int8 would get some big savings for power and throughput. NVIDIA is playing this game too. At Google/FB scale tweaks to improve cost may make sense but it doesnt seem like architecturally speaking there is some major design decision being left on the table anymore (I'd love to hear a specific counterpoint though!).

In terms of inference, there can be some big power savings putting the weights next to compute. This isn't rocket science and folks have been doing this for a while. In a strange way it can sometimes be more efficient to use a CPU, with fat caches.

Re: The Looming Battle Over AI Chips

#64

Post by joshuamorton »

emcq wrote: post #63
joshuamorton wrote: post #51

Earlier quoted context omitted.

I'm a little confused here, are you saying that ML ASICs can't beat compute per $ of GPUs? That seems, on its face, to be a ridiculous assertion, so I'm confused where I'm misunderstanding you.

At a high level there is a design tradeoff where you put your transistors for a given chip. For a dense linear algebra/tensor processor, it basically comes down to using your transistors for memory or compute. GPUs (and DSPs) historically are way on the compute side. You get kilobytes of on chip memory, and really fat parallel buses to off chip RAM. On the other end, you have some chips that put more memory near the…

>At Google/FB scale tweaks to improve cost may make sense but it doesnt seem like architecturally speaking there is some major design decision being left on the table anymore (I'd love to hear a specific counterpoint though!).

I'd expect if I knew any, they'd be under NDA. I'll just point out that a GPU, even one with specific "ML cores" as NVIDIA calls them, is going to have a bunch of silicon that is being used inefficiently (for more "conventional" GPU uses). There's room for cost saving there. Perhaps NVIDIA eventually moves into that space and produces ML-only chips, but they don't appear to be heading in that direction yet.

Re: The Looming Battle Over AI Chips

#65

Post by nl »

oneshot908 wrote: post #15
joe_the_user wrote: post #12

Nvidia, moreover, increasingly views its software for programming its chips, called CUDA, as a kind of vast operating system that would span all of the machine learning in the world, an operating system akin to what Microsoft (MSFT) was in the old days of PCs. Yeah, nVidia throwing it's weight around in terms of requiring that data centers pay more to use cheap consumer gaming chips may turn out to backfire and certa…

It probably won't. For every oppressive move NVDA has made so far, there has been a swarm of low-information technophobe MBA sorts who eat their computational agitprop right up, some of them even fashion themselves as data scientists. More likely, NVDA continues becoming the Oracle of AI that everyone needs and everyone hates.

This is complete nonsense, and since you’ve said it twice in this thread I’ll respond.

The reason NVidia is used in deep learning is because the frameworks give the best performance on it.

The reason they give good performance on it is because CUDA an CuDNN (which is even more important but I don’t see mention of) work really well and give better performance than anything else.

There are no MBA, just lots of grad students trying anything to get better speed.

Re: The Looming Battle Over AI Chips

#66

Post by etaioinshrdlu »

joe_the_user wrote: post #61
Symmetry wrote: post #58

Earlier quoted context omitted.

Those are some really clever ways to make sure that all the threads in your program are executing the same instruction, but it doesn't get around the problem. Thanks for linking that video, though.

The key of the Dietz system (MOG) is that the native code that the GPU runs is a bytecode interpreter. Bytecode "instruction pointer" together with other data is just data in registers and memory that's interpreted by the native code interpreter. So for each thread, the instruction pointer can point at a different command - the interpreter runs the same instructions but the results are different. So effectively you a…

This is more or less what I'm talking about. I wonder what possibilities lie with using the huge numerical computation available on a GPU applied to predictive parts of a CPU, such as memory prefetch prediction, branch prediction, etc.

Not totally dissimilar to the thinking behind NetBurst which seemed to be all about having a deep pipeline and keeping it fed with quality predictions.

Re: The Looming Battle Over AI Chips

#67

Post by keldaris »

arca_vorago wrote: post #40
oneshot908 wrote: post #15

Earlier quoted context omitted.

It probably won't. For every oppressive move NVDA has made so far, there has been a swarm of low-information technophobe MBA sorts who eat their computational agitprop right up, some of them even fashion themselves as data scientists. More likely, NVDA continues becoming the Oracle of AI that everyone needs and everyone hates.

So is OpenCL dead? Because that's how everyone is talking. The tools you choose, and their licensing, matters!

OpenCL isn't dead, if you write your code from scratch you can use it just fine and match CUDA performance. In my experience, OpenCL has two basic issues.

The first is the ecosystem. Nvidia went to great lengths to provide well optimized libraries built on top of CUDA that supply things people care about - deep learning stuff, dense as well as sparse linear algebra, etc. There's nothing meaningfully competitive on the OpenCL side of things.

The second is user friendliness of the API and the implementations. OpenCL is basically analogous to OpenGL in terms of design, it's a verbose annoying C API with huge amounts of trivial boilerplate. By contrast, CUDA supports most of the C++ convenience features relevant in this problem space, has decent tools, IDE and debugger integration, etc.

Neither of these issues is necessary a dealbreaker if you're willing to invest the effort, but choosing OpenCL over CUDA requires prioritizing portability over user friendliness, available libraries and tooling. As a consequence, not many people choose OpenCL and the dominance of CUDA continues to grow. Unfortunately, I don't see that changing in the near future.

Re: The Looming Battle Over AI Chips

#68

Post by sanxiyn »

joshuamorton wrote: post #64
emcq wrote: post #63

Earlier quoted context omitted.

At a high level there is a design tradeoff where you put your transistors for a given chip. For a dense linear algebra/tensor processor, it basically comes down to using your transistors for memory or compute. GPUs (and DSPs) historically are way on the compute side. You get kilobytes of on chip memory, and really fat parallel buses to off chip RAM. On the other end, you have some chips that put more memory near the…

>At Google/FB scale tweaks to improve cost may make sense but it doesnt seem like architecturally speaking there is some major design decision being left on the table anymore (I'd love to hear a specific counterpoint though!). I'd expect if I knew any, they'd be under NDA. I'll just point out that a GPU, even one with specific "ML cores" as NVIDIA calls them, is going to have a bunch of silicon that is being used ine…

NVIDIA is heading in that direction: see NVDLA. http://nvdla.org/

Re: The Looming Battle Over AI Chips

#69

Post by joe_the_user »

etaioinshrdlu wrote: post #66
joe_the_user wrote: post #61

Earlier quoted context omitted.

The key of the Dietz system (MOG) is that the native code that the GPU runs is a bytecode interpreter. Bytecode "instruction pointer" together with other data is just data in registers and memory that's interpreted by the native code interpreter. So for each thread, the instruction pointer can point at a different command - the interpreter runs the same instructions but the results are different. So effectively you a…

This is more or less what I'm talking about. I wonder what possibilities lie with using the huge numerical computation available on a GPU applied to predictive parts of a CPU, such as memory prefetch prediction, branch prediction, etc. Not totally dissimilar to the thinking behind NetBurst which seemed to be all about having a deep pipeline and keeping it fed with quality predictions.

I'm not sure if your idea in particular is possible but who knows. There may be fundamental limits to speeding up computation based speculative look-ahead not matter how many parallel tracks you have and it may run into memory through-put issues.

But take a look at the MOG code and see what you can do.

Check out H. Dietz' stuff. Links above.

Re: The Looming Battle Over AI Chips

#70

Post by sanxiyn »

taeric wrote: post #52
jacksmith21006 wrote: post #29

The new Google Speech solution is the perfect example on why Google had to do their own silicon. Doing speech with 16k samples a second through a NN and keep at a reasonable cost is really, really difficult. The old way was far more power efficient and if you are going to use this new technique which gets you a far better result and do it at a reasonable cost you have to go all the way down into the silicon. Here lis…

This somewhat blows my mind. Yes, it is impressive. However, the work that Nuance and similar companies used to do are still competitive, just not getting near the money and exposure. I remember over a decade ago, they even had mood analysis they could apply to listening to people. Far from new. Is it truly more effective or efficient nowadays? Or just getting marketed by companies you've heard of?

It is truly better. Objective metrics (such as word error rate) don't lie. You can argue whether it makes sense to use, say, 100x compute to get 2x less error, but that's a different argument; I don't think anyone is really disputing improved quality.
Post reply on HN