Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google's First Tensor Processing Unit: Architecture

thechipletter.substack.com

Post reply on HN
Print view View original

151–160 of 197 posts

Re: Google's First Tensor Processing Unit: Architecture

#151

Post by Symmetry »

IncreasePosts wrote: post #11

Sigh...learning about TPUs a decade ago made me invest heavily in $GOOG for the coming AI revolution...got that one 100% wrong. +400% over 10 years isn't bad but I can't help but feel shortchanged seeing nvidia/etc

The potential usefulness of things like TPUs actually made me invest in Broadcomm, which helped Google design them and could potentially help Amazon or however else design their equivalents. But I'm also long NVidia and a half dozen other companies with AI exposure while still keeping most of my money in index funds.

Re: Google's First Tensor Processing Unit: Architecture

#152

Post by totallyanon »

bartwr wrote: post #135
Laremere wrote: post #88

Earlier quoted context omitted.

The way I see, NVidia only has a few advantages ordered from most important to least: 1. Reserved fab space. 2. Highly integrated software. 3. Hardware architecture that exists today. 4. Customer relationships. but all of these aspects are weak in one way or another: For #1, fab space is tight, and NVidia can strangle its consumer GPU market if it means selling more AI chips at a higher price. This advantage is gone…

Seems you have not worked with ML workloads, but base your comment on "internet wisdom", or worse, business analysts (I am sorry if that's inaccurate). On GPUs, ML "just works" (inference and training) and are always order of magnitude faster than whatever CPU you have. TPUs work very well for some model architectures (old ones that they were optimized and designed for) and on some novel others can be actually slower…

[dead]

Re: Google's First Tensor Processing Unit: Architecture

#153

Post by Kelteseth »

refulgentis wrote: post #146
Kelteseth wrote: post #126

Earlier quoted context omitted.

If AMD fixes or open sources their proprietary firmware blob[0]. Geohot streamed all weekend on Twitch, reverse engineering the AMD firmware. It was quite entertaining learning about how that low level hardware firmware works[1] and his rants about AMD of course. [0] https://www.phoronix.com/news/Tinybox-Radeon-Again-UMR [1] https://www.twitch.tv/georgehotz

Geohot doesn't know what he's talking about and I'm kinda ashamed to see this lazy thinking leak onto HN. There was an article a couple weeks back on AMD open sourcing drivers in the Linux kernel tree that you should look into.

Care to explain a bit more? His rant was about the firmware having crashes not the Linux driver.

Re: Google's First Tensor Processing Unit: Architecture

#154

Post by puppymaster »

sevagh wrote: post #138
bartwr wrote: post #135

Earlier quoted context omitted.

Seems you have not worked with ML workloads, but base your comment on "internet wisdom", or worse, business analysts (I am sorry if that's inaccurate). On GPUs, ML "just works" (inference and training) and are always order of magnitude faster than whatever CPU you have. TPUs work very well for some model architectures (old ones that they were optimized and designed for) and on some novel others can be actually slower…

Also, it's disingenuous to say "there's only 4 things you need to beat NVIDIA" when each of the 4 is an enormous undertaking.

not to mention every not-so-serious, inference heavy ML developers just want something to work to deliver to client. That itself is a semi-moat.

Re: Google's First Tensor Processing Unit: Architecture

#155

Post by refulgentis »

Kelteseth wrote: post #153
refulgentis wrote: post #146

Earlier quoted context omitted.

Geohot doesn't know what he's talking about and I'm kinda ashamed to see this lazy thinking leak onto HN. There was an article a couple weeks back on AMD open sourcing drivers in the Linux kernel tree that you should look into.

Care to explain a bit more? His rant was about the firmware having crashes not the Linux driver.

Firmware crashes => days long "open source it and I'll fix it. no? why does AMD hate its customers?"

I got an appointment and have exactly one minute till I have to leave, apologies for brevity: they can't open source the full driver because then they'd have to release HDMI spec stuff that the consortium says they can't. (I don't support any of that, my only intent is to communicate George isn't really locked in here when he starts casting aspersions or claiming AMD doesn't care)

Re: Google's First Tensor Processing Unit: Architecture

#156

Post by kkielhofner »

puppymaster wrote: post #154
sevagh wrote: post #138

Earlier quoted context omitted.

Also, it's disingenuous to say "there's only 4 things you need to beat NVIDIA" when each of the 4 is an enormous undertaking.

not to mention every not-so-serious, inference heavy ML developers just want something to work to deliver to client. That itself is a semi-moat.

It's been talked to death but non-CUDA implementations have their challenges regardless of use case. That's what first-mover advantage and > 15 years of investment by Nvidia in their overall ecosystem will do for you.

But support for production serving of inference workloads outside of CUDA is universally dismal. This is where I spend most of my time and compared to CUDA anything else is non-existent or a non-starter unless you're all-in on packaged API driven Google/Amazon/etc tooling utilizing their TPUs (or whatever). The most significant vendor/cloud lock-in I think I've ever seen.

Efficient and high-scale serving of inference workloads is THE thing you need to do to serve customers and actually have a chance at ever making any money. It's shocking to me that Nvidia/CUDA has a complete stranglehold on this obvious use case.

Re: Google's First Tensor Processing Unit: Architecture

#157

Post by bionhoward »

nl wrote: post #29

On the podcast interview now Groq CEO Jonathon Ross did[1] he talked about the creation of the original TPUs (which he built at Google). Apparently originally it was a FPGA he did in his 20% time because he sat near the team who was having inference speed issues. They got it working, then Jeff Dean did the math and the decided to do an ASIC. Now of course Google should spin off the TPU team as a separate company. It'…

Speaking of which, mega props to Groq, they really are awesome, so many startups launch with bullshit and promises, but Groq came to the scene with something awesome already working, which is reason enough to love them. I really respect this company and I say that extremely never-often.

Re: Google's First Tensor Processing Unit: Architecture

#158

Post by sevagh »

kkielhofner wrote: post #156
puppymaster wrote: post #154

Earlier quoted context omitted.

not to mention every not-so-serious, inference heavy ML developers just want something to work to deliver to client. That itself is a semi-moat.

It's been talked to death but non-CUDA implementations have their challenges regardless of use case. That's what first-mover advantage and > 15 years of investment by Nvidia in their overall ecosystem will do for you. But support for production serving of inference workloads outside of CUDA is universally dismal. This is where I spend most of my time and compared to CUDA anything else is non-existent or a non-starter…

A great summary of how unserious NVIDIA's competitors are is how long it took AMD's flagship consumer/retail GPU, the 7900 XT[X], to gain ROCm support.

That's quite literally unacceptable.

Re: Google's First Tensor Processing Unit: Architecture

#159

Post by kkielhofner »

KeplerBoy wrote: post #128
Kelteseth wrote: post #126

Earlier quoted context omitted.

If AMD fixes or open sources their proprietary firmware blob[0]. Geohot streamed all weekend on Twitch, reverse engineering the AMD firmware. It was quite entertaining learning about how that low level hardware firmware works[1] and his rants about AMD of course. [0] https://www.phoronix.com/news/Tinybox-Radeon-Again-UMR [1] https://www.twitch.tv/georgehotz

Geohot is wrangling with unsupported consumer hardware. The datacenter stuff is on a different architecture and driver stack. The number one supercomputer on the top500 list (frontier at ORNL) is based on AMD GPUs and AMD is probably more invested in supporting that.

I work with Frontier and ORNL/OLCF. They have had and continue to have issues with AMD/ROCm but yes, they do of course get excellent support from AMD. The entire team at OLCF is incredible as well (obviously) and they do amazing work.

Frontier certainly has some unique quirks but the documentation is online[0] and most of these quirks are inherent to the kinds of fundamental issues you'll see on any system in the space (SLURM, etc).

However, most of the issues are fundamentally ROCm and you'll run into them on any MIxxx anywhere. I run into them frequently with supported and unsupported consumer gear all the way up.

[0] - https://docs.olcf.ornl.gov/systems/frontier_user_guide.html

Re: Google's First Tensor Processing Unit: Architecture

#160

Post by disgruntledphd2 »

varjag wrote: post #119
exitheone wrote: post #111

Earlier quoted context omitted.

The cost per ad is still astronomically different between search ads and LLMs

There could be an opposite avenue: ad-free Google Premium subscription with AI chat as a crown jewel. An ultimate opportunity to diversify from ad revenue.

There's not enough money in it, as Google's scale.

Especially because the people who'd pay for Premium tend to be the most prized people from an advertiser perspective.

And most people won't pay, under any circumstances, but they will click on ads which make Google money.

Post reply on HN