Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google's First Tensor Processing Unit: Architecture

thechipletter.substack.com

Post reply on HN
Print view View original

191–197 of 197 posts

Re: Google's First Tensor Processing Unit: Architecture

#191

Post by sigmoid10 »

latchkey wrote: post #186
sigmoid10 wrote: post #167

Earlier quoted context omitted.

>On GPUs, ML "just works" If you had worked with ML, you'd know that this is not true. It's actually more like the opposite. It also has nothing to do with the chips themselves. Things don't magically work "because GPU", they work because manufacturers spend the time getting their drivers and ecosystems right. That's why for example noone is using AMD GPUs for ML, despite them offering more compute per dollar on pape…

> That's why for example noone is using AMD GPUs for ML You're right, they are behind, but to say that nobody is using it, is not truthful. AMD HPC clusters are being used [0] and [1] for AI/ML. The larger issue is that AMD has only been building HPC clusters for the last period of time. Now, with the release of MI300x, we have Azure and Oracle coming online with them now. Disclosure, my business is also building a M…

>AMD HPC clusters are being used [0] and [1] for AI/ML.

Funny how you can immediately tell when the business people made these decisions and not the tech people. This is exactly what I would have expected from an organization like the Navy. On paper it does sound great and the Navy bean counters probably loved this. But they are in for a rough awakening.

Re: Google's First Tensor Processing Unit: Architecture

#192

Post by latchkey »

sigmoid10 wrote: post #191
latchkey wrote: post #186

Earlier quoted context omitted.

> That's why for example noone is using AMD GPUs for ML You're right, they are behind, but to say that nobody is using it, is not truthful. AMD HPC clusters are being used [0] and [1] for AI/ML. The larger issue is that AMD has only been building HPC clusters for the last period of time. Now, with the release of MI300x, we have Azure and Oracle coming online with them now. Disclosure, my business is also building a M…

>AMD HPC clusters are being used [0] and [1] for AI/ML. Funny how you can immediately tell when the business people made these decisions and not the tech people. This is exactly what I would have expected from an organization like the Navy. On paper it does sound great and the Navy bean counters probably loved this. But they are in for a rough awakening.

As far as I can tell, the only rough awakening is that they paid $25m in 2023, that costs a fraction of that today, for even better performance.

In a few months, my own cluster will be nearly 2x that size, with better networking, and we aren't spending anywhere near $25m.

Disclosure: building my own supercomputer business around AMD hardware

Re: Google's First Tensor Processing Unit: Architecture

#195

Post by nl »

rurban wrote: post #114
nl wrote: post #103

Earlier quoted context omitted.

Yes, but they don't have the custom kernels that CUDA has. TPUs do have some!

They have Vulcan, which is cross-compatible. And AMD has ROCm. pytorch is standard and pytorch has ROCm support. And the Google TPU v5 also has pytorch support. We do have a couple of H100's, but I'd love to replace them with AMD's

Vulcan is a driver-level API. It competes with DirectX and OpenGL.

CUDA is a language you write kernels. It competes with OpenAI's Triton language.

Here's what CUDA looks like: https://github.com/tspeterkim/flash-attention-minimal/blob/m...

This is what Triton looks like: https://triton-lang.org/main/getting-started/tutorials/06-fu...

By contrast Vulcan looks like this: https://github.com/KhronosGroup/Vulkan-Samples/blob/main/sam...

(It's true to some extent that maybe you could use Vulcan shaders to write deep learning kernels, maybe? I'm not aware of anyone doing it though)

Re: Google's First Tensor Processing Unit: Architecture

#196

Post by sigmoid10 »

latchkey wrote: post #192
sigmoid10 wrote: post #191

Earlier quoted context omitted.

>AMD HPC clusters are being used [0] and [1] for AI/ML. Funny how you can immediately tell when the business people made these decisions and not the tech people. This is exactly what I would have expected from an organization like the Navy. On paper it does sound great and the Navy bean counters probably loved this. But they are in for a rough awakening.

As far as I can tell, the only rough awakening is that they paid $25m in 2023, that costs a fraction of that today, for even better performance. In a few months, my own cluster will be nearly 2x that size, with better networking, and we aren't spending anywhere near $25m. Disclosure: building my own supercomputer business around AMD hardware

The best I can say is that my thoughts and prayers go to the ML engineers who will actually have to deal with this. Those companies literally couldn't pay me enough to put up with it. They will likely only attract people who care about the salary and the position instead of getting things done. I've seen it with other colleagues before. These numbers of yours are completely worthless without someone who is willing to put in 5 times the work for the same or worse results.

Re: Google's First Tensor Processing Unit: Architecture

#197

Post by latchkey »

sigmoid10 wrote: post #196
latchkey wrote: post #192

Earlier quoted context omitted.

As far as I can tell, the only rough awakening is that they paid $25m in 2023, that costs a fraction of that today, for even better performance. In a few months, my own cluster will be nearly 2x that size, with better networking, and we aren't spending anywhere near $25m. Disclosure: building my own supercomputer business around AMD hardware

The best I can say is that my thoughts and prayers go to the ML engineers who will actually have to deal with this. Those companies literally couldn't pay me enough to put up with it. They will likely only attract people who care about the salary and the position instead of getting things done. I've seen it with other colleagues before. These numbers of yours are completely worthless without someone who is willing to…

People choose jobs and tools for a variety of reasons. I don't feel the need to cast judgement on them over it.

The numbers I gave aren't worthless, nor does it take 5x the amount of work. I also don't think that going with a single source for hardware for all of AI is very smart either, especially given the fact that there are serious supply shortages from that single vendor. No fortune 100 would put all their eggs in one basket and even if it was 5x the work, it is worth it.

Post reply on HN