Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intel CEO: 'The entire industry is motivated to eliminate the CUDA market'

tomshardware.com

Post reply on HN
Print view View original

171–180 of 380 posts

Re: Intel CEO: 'The entire industry is motivated to eliminate the CUDA market'

#171

Post by raphlinus »

jauntywundrkind wrote: post #167
raphlinus wrote: post #156

Earlier quoted context omitted.

Vulkan 1.3 has pointers, thanks to buffer device address[1]. It took a while to get there, and earlier pointer support was flawed. I also don't know of any major applications that use this. Modern Vulkan is looking pretty good now. Cooperative matrix multiplication has also landed (as a widely supported extension), and I think it's fair to say it's gone past OpenCL. Whether we get significant adoption of all this I t…

Is IREE the main runtime doing Vulkan or are there others? Who should we be listening to (oh wise @raphlinus)? It's been awesome seeing folks like Keras 3.0 kicking out broad Intercompatibility across JAX, TF, Pytorch, powered by flexible executuon engines. Looking forward to seeing more Vulkan based runs getting socialized benchmarked & compared. https://news.ycombinator.com/item?id=38446353

The two I know of are IREE and Kompute[1]. I'm not sure how much momentum the latter has, I don't see it referenced much. There's also a growing body of work that uses Vulkan indirectly through WebGPU. This is currently lagging in performance due to lack of subgroups and cooperative matrix mult, but I see that gap closing. There I think wonnx[2] has the most momentum, but I am aware of other efforts.

[1]: https://kompute.cc/

[2]: https://github.com/webonnx/wonnx

Re: Intel CEO: 'The entire industry is motivated to eliminate the CUDA market'

#172

Post by flakiness »

ndneighbor wrote: post #165
lacker wrote: post #66

Earlier quoted context omitted.

The funny thing to me is that so much of the "AI software ecosystem" is just PyTorch. You don't need to develop some new framework and make it popular. You don't need to support a zillion end libraries. Just literally support PyTorch. If PyTorch worked fine on Intel GPUs, a lot of people would be happy to switch.

OneAPI isn't bad for PyTorch, the performance isn't there yet but you can tell it's an extremely top priority for Intel.

Intel has to do it by themselves. NVIDIA just lets Meta/OpenAI/Google engineers do it for them. Such a handicapped fight.

Re: Intel CEO: 'The entire industry is motivated to eliminate the CUDA market'

#173

Post by shihab »

singhrac wrote: post #103
shihab wrote: post #83

Earlier quoted context omitted.

But you can't support Pytorch without a proper foundation in place. They don't need to support zillion _end_ libraries, sure, but they do need to have at least a very good set of standard libraries, equivalent of Cublas, Curand etc. And they don't. My work recently had me working with rocRAND (Rocm's answer to Curand). It was frankly pretty bad- the design, performance (50% slower in places that don't make any sense…

Generating random numbers is a bit complicated! I wrote some of the samplers in Pytorch (probably replaced by now) and some of the underlying pseudo-random algorithms that work correctly in parallel are not exactly easy... running the same PRNG with the same seed on all your cores will produce the same result, which is probably NOT what you want from your API. But, to be honest, it's not that hard either. I'm surpris…

> Generating random numbers is a bit complicated!

I know! I just wrote a whole paper and published a library on this!

But really, perhaps not as much as many from outside might think. The core of a Philox implementation can be around 50 lines of C++ [1], with all the bells and whistles maybe around 300-400. That implementation's performance equals CuRAND's , sometimes even surpasses it! (the API is designed to avoid maintaining any rng states on device memory, something curand forces you to do).

> running the same PRNG with the same seed on all your cores will produce the same result

You're right. Solution here is to utilize multiple generator objects, one per thread, ensuring each produces statistically independent random streams. Some good algorithms (Philox for example), allow you to use any set of unique values as seeds for your threads (e.g. thread id).

[1] https://github.com/msu-sparta/OpenRAND/blob/main/include/ope...

Re: Intel CEO: 'The entire industry is motivated to eliminate the CUDA market'

#174

Post by 2OEH8eoCRo0 »

buildbot wrote: post #33

As another commenter said, it's CUDA. Intel and AMD and whoever can turn out chips reasonably fast, but nobody gets that it's the software and ecosystem. You have to out-compete the ecosystem. You can pick up a used Mi100 that performs almost like an A100 for 5x less money on eBay for example. Why is it 5x less? Because the software incompatibilities mean you'll spend a ton of time getting it to work compared to an N…

> the software incompatibilities mean you'll spend a ton of time getting it to work compared to an Nvidia GPU.

Leverage LLMs to port the SW

Re: Intel CEO: 'The entire industry is motivated to eliminate the CUDA market'

#175

Post by lmm »

latchkey wrote: post #77
UncleOxidant wrote: post #62

Earlier quoted context omitted.

This right here. Until Intel (and/or AMD) get serious about the software side and actually invest the money CUDA isn't going anywhere. Intel will make noises about various initiatives in that direction and then a quarter or two later they'll make big cuts in those divisions. They need to make a multi-year commitment and do some serious hiring (and they'll need to raise their salaries to market rates to do this) if th…

Literally, every single announcement (and action) coming out of AMD these days is that they are serious about the software. I don't see any reason at this point to doubt them. The larger issue is that they need to fix the access to their high end GPUs. You can't rent a MI250... or even a MI300x (yet, I'm working on that myself!). But that said, you can't rent an H100 either... there are none available.

> Literally, every single announcement (and action) coming out of AMD these days is that they are serious about the software. I don't see any reason at this point to doubt them.

They're having to announce it so much because people are rightly sceptical. Talk is cheap, and their software has sucked for years. Have they given concrete proof of their commitment, e.g. they've spent X dollars or hired Y people to work on it (or big names Z and W)?

Re: Intel CEO: 'The entire industry is motivated to eliminate the CUDA market'

#176

Post by zozbot234 »

raphlinus wrote: post #171
jauntywundrkind wrote: post #167

Earlier quoted context omitted.

Is IREE the main runtime doing Vulkan or are there others? Who should we be listening to (oh wise @raphlinus)? It's been awesome seeing folks like Keras 3.0 kicking out broad Intercompatibility across JAX, TF, Pytorch, powered by flexible executuon engines. Looking forward to seeing more Vulkan based runs getting socialized benchmarked & compared. https://news.ycombinator.com/item?id=38446353

The two I know of are IREE and Kompute[1]. I'm not sure how much momentum the latter has, I don't see it referenced much. There's also a growing body of work that uses Vulkan indirectly through WebGPU. This is currently lagging in performance due to lack of subgroups and cooperative matrix mult, but I see that gap closing. There I think wonnx[2] has the most momentum, but I am aware of other efforts. [1]: https://kom…

How feasible would it be to target Vulkan 1.3 or such from standard SYCL (as first seen in Sylkan, for earlier Vulkan Compute)? Is it still lacking the numerical properties for some math functions that OpenCL and SYCL seem to expect?

Re: Intel CEO: 'The entire industry is motivated to eliminate the CUDA market'

#177

Post by EdSharkey »

rqtwteye wrote: post #147
ryanjshaw wrote: post #44

Earlier quoted context omitted.

I've got bad news for you. AD, Office, SharePoint, Teams, BI, etc. are becoming unified in the cloud around Power Platform. Any new business problem in a Microsoft shop is going to go with a Power Platform/Teams/SharePoint Online solution as first prize. Personally, I like it, even if it has some annoying challenges. I've not seen anything else come close in terms of rapid application development, integration and dep…

But don't forget: It's Sharepoint all the way down.

And on the way down, there was an unholy congress between SharePoint and OneDrive to create a new eldritch horror.

Re: Intel CEO: 'The entire industry is motivated to eliminate the CUDA market'

#178

Post by raphlinus »

zozbot234 wrote: post #176
raphlinus wrote: post #171

Earlier quoted context omitted.

The two I know of are IREE and Kompute[1]. I'm not sure how much momentum the latter has, I don't see it referenced much. There's also a growing body of work that uses Vulkan indirectly through WebGPU. This is currently lagging in performance due to lack of subgroups and cooperative matrix mult, but I see that gap closing. There I think wonnx[2] has the most momentum, but I am aware of other efforts. [1]: https://kom…

How feasible would it be to target Vulkan 1.3 or such from standard SYCL (as first seen in Sylkan, for earlier Vulkan Compute)? Is it still lacking the numerical properties for some math functions that OpenCL and SYCL seem to expect?

That's a really good question. I don't know enough about SYCL to be able to tell you the answer, but I've heard rumblings that it may be the thing to watch. I think there may be some other limitations, for example SYCL 2020 depends on unified shared memory, and that is definitely not something you can depend on in compute shader land (in some cases you can get some of it, for example with resizable BAR, but it depends).

In researching this answer, I came across a really interesting thread[1] on diagnosing performance problems with USM in SYCL (running on AMD HIP in this case). It's a good tour of why this is hard, and why for the vast majority of users it's far better to just use CUDA and not have to deal with any of this bullshit - things pretty much just work.

When targeting compute shaders, you pretty much have to manage buffers manually, and also do copying between host and device memory explicitly (when needed - on hardware such as Apple Silicon, you prefer to not copy). I personally don't have a problem with this, as I like things being explicit, but it is definitely one of the ergonomic advantages of modern CUDA, and one of the reasons why fully automated conversion to other runtimes is not going to work well.

[1]: https://stackoverflow.com/questions/76700305/4000-performanc...

Re: Intel CEO: 'The entire industry is motivated to eliminate the CUDA market'

#179

Post by gamblor956 »

AnthonyMouse wrote: post #96
foobiekr wrote: post #64

Earlier quoted context omitted.

It's not just Intel. Open initiatives and consortiums (the phase two of the same) are always the losers ganging up hoping that it will give them the leg up they don't have. If you're older you'll have seen this play out over and over in the industry - the history of Unix vs. Windows NT from the 1990s was full of actions like this, networking is going through it again for the nth time (this time with UltraEthernet) an…

You're just listing the ones that didn't knock it out of the park. TCP/IP completely displaced IPX to the point that most people don't even remember what it was. Nobody uses WINS anymore, even Microsoft uses DNS. It's rare to find an operating system that doesn't implement the POSIX API. The past is littered with the corpses of proprietary technologies displaced by open standards. Because customers don't actually wan…

Windows still uses WINS and NetBIOS when DNS is unavailable.

Re: Intel CEO: 'The entire industry is motivated to eliminate the CUDA market'

#180

Post by raphlinus »

Const-me wrote: post #169
raphlinus wrote: post #157

Earlier quoted context omitted.

I understand why you've picked D3D11, but people have to understand that comes with serious limitations. There are no subgroups, which also means no cooperative matrix multiplication ("tensor cores"). For throughput in machine learning inference in particular, there's no way D3D11 can compete with either CUDA or a more modern compute shader stack, such as one based on Vulkan 1.3.

> no subgroups Indeed, in D3D they are called “wave intrinsics” and require D3D12. But that’s IMO a reasonable price to pay for hardware compatibility. > no cooperative matrix multiplication Matrix multiplication compute shader which uses group shared memory for cooperative loads: https://github.com/Const-me/Cgml/blob/master/Mistral/Mistral... > tensor cores When running inference on end-user computers, for many prac…

Sure. It's a tradeoff space. Gain portability and ergonomics, lose throughput. For applications that are throttled by TOPS at low precisions (ie most ML inferencing) then the performance drop from not being able to use tensor cores is going to be unacceptable. Glad you found something that works for you, but it certainly doesn't spell the end of CUDA.
Post reply on HN