Google's First Tensor Processing Unit: Architecture • Hacker News Board

Re: Google's First Tensor Processing Unit: Architecture

#161

Post by Oioioioiio » Tue, Mar 26, 2024, 1:53 PM UTC

Laremere wrote: post #88

nl wrote: post #29

On the podcast interview now Groq CEO Jonathon Ross did[1] he talked about the creation of the original TPUs (which he built at Google). Apparently originally it was a FPGA he did in his 20% time because he sat near the team who was having inference speed issues. They got it working, then Jeff Dean did the math and the decided to do an ASIC. Now of course Google should spin off the TPU team as a separate company. It'…

The way I see, NVidia only has a few advantages ordered from most important to least: 1. Reserved fab space. 2. Highly integrated software. 3. Hardware architecture that exists today. 4. Customer relationships. but all of these aspects are weak in one way or another: For #1, fab space is tight, and NVidia can strangle its consumer GPU market if it means selling more AI chips at a higher price. This advantage is gone…

Nvidia has so much software behind all of this, your list is a tremendes understatement.

Alone how many internal ML things nvidia builds helps them tremendesly to understand the market (what does the market need).

And they use their inventions themselves.

'only has a few' = 'has a handful easy to list but with huge implications which are not easily matched by amd or intel right now'

Re: Google's First Tensor Processing Unit: Architecture

#162

Post by Rinzler89 » Tue, Mar 26, 2024, 2:05 PM UTC

sofixa wrote: post #131

Rinzler89 wrote: post #124

Earlier quoted context omitted.

>and the batteries only lasted like four hours Still more than the OG Steam Deck today :)

Vastly depends on the game played and the settings. In a plane (so airplane mode, with Bluetooth headset) I played Hitman Absolution for 3 hours and still had 50%+ of the battery left. It was on minimal brightness because it was dark and didn't need more, but still.

Yeah, no need to take a (semi)joke literally and go all technical to debunk it. Though without optimizations, battery life on the deck was lucky to hit 2h at first before valve brought in updates and people learned they had to cap resolution and FPS to increase battery life.

Re: Google's First Tensor Processing Unit: Architecture

#163

Post by _ea1k » Tue, Mar 26, 2024, 2:07 PM UTC

michaelt wrote: post #140

foota wrote: post #67

Earlier quoted context omitted.

How many people are out there buying H100s for their personal use?

Ah, but part of the reason for CUDA's success is that the open source developer who wants to run unit tests or profile their kernel can pick up a $200 card. That PhD student with a $2000 budget can pick up a card. Academic lab with $20,000 for a beefy server, or tiny cluster? nvidia will take their money. And that's all fixed capital expenditure - there's no risk a code bug or typo by an inexperienced student will le…

I'm really shocked at how dependent companies have become on the cloud offerings. Want a GPU? Those are expensive, lets just rent on Amazon and then complain about operational costs!

I've noticed this at companies. Yeah, the cloud is expensive, but you have a data center, and a few servers with RTX 3090s aren't expensive. A lot of research workloads can run on simple, cheap hardware.

Even older Nvidia P40s are still useful.

Re: Google's First Tensor Processing Unit: Architecture

#164

Post by varjag » Tue, Mar 26, 2024, 2:25 PM UTC

disgruntledphd2 wrote: post #160

varjag wrote: post #119

Earlier quoted context omitted.

There could be an opposite avenue: ad-free Google Premium subscription with AI chat as a crown jewel. An ultimate opportunity to diversify from ad revenue.

There's not enough money in it, as Google's scale. Especially because the people who'd pay for Premium tend to be the most prized people from an advertiser perspective. And most people won't pay, under any circumstances, but they will click on ads which make Google money.

YouTube does it, at Google scale. And these same people do pay $20/mo for ChatGPT anyway.

Re: Google's First Tensor Processing Unit: Architecture

#165

Post by IX-103 » Tue, Mar 26, 2024, 2:35 PM UTC

Symmetry wrote: post #147

Karrot_Kream wrote: post #99

Earlier quoted context omitted.

Google gets much more scrutiny then smaller companies so it's understandable to be worried. Pretty much any small mistake of theirs turns into clickbait on here and the other tech news sites and you get hundreds of comments about how evil Big Tech is. Of course it's their own fault that their PR hews negative so frequently but still it's understandable why they were so shy.

It's understandable that people at Google are worried because it's likely very unpleasant to see critical articles and tweets about something you did. But that isn't really bad for Google's business in any of the ways that losing to someone on AI would be.

Google is constantly being sued for nearly everything they do. They create a Chrome Incognito mode like Firefox's private browsing mode and they get sued. They start restricting App permissions on Android, sued. Adding a feature where Google maps lets you select the location of your next appointment as a destination in a single click, sued (that's leveraging your calendar monopoly to improve your map app).

Google has it's hands in so many fields that any change they make that disrupts the status-quo brings down antitrust investigations and lawsuits.

That's the reason why Firefox and Safari dropping support for 3rd party cookies gets a yawn from regulators while Google gets pinned between the CMA wanting to slow down or stop 3rd party cookies deprecation to prevent disrupting the ads market and the ICO wanting Google to drop support yesterday.

This is not about bad press or people feeling bad about news articles. Google has been hit by billion dollar fines in the past and has become hesitant to do anything.

Where smaller companies can take the "Elon Musk" route and just pay fines and settle lawsuits as just the cost of doing business, Google has become an unwieldy juggernaut unable to move out of fear of people complaining and taking another pound of flesh. To be clear, I don't agree with a strategy of ignoring inconvenient regulations, but Google's excess of caution has severely limited their ability to innovate. But given previous judgements against Google, I can't exactly say that they're wrong to do so. Even Google can only pay so many multi-billion dollar fines before they have to close shop, and I can't exactly say the world would be better off if that happened.

Re: Google's First Tensor Processing Unit: Architecture

#166

Post by michaelt » Tue, Mar 26, 2024, 2:37 PM UTC

Symmetry wrote: post #147

Karrot_Kream wrote: post #99

Earlier quoted context omitted.

Google gets much more scrutiny then smaller companies so it's understandable to be worried. Pretty much any small mistake of theirs turns into clickbait on here and the other tech news sites and you get hundreds of comments about how evil Big Tech is. Of course it's their own fault that their PR hews negative so frequently but still it's understandable why they were so shy.

It's understandable that people at Google are worried because it's likely very unpleasant to see critical articles and tweets about something you did. But that isn't really bad for Google's business in any of the ways that losing to someone on AI would be.

That's true for google, sure. But what about individual workers and managers at google?

You can push things forward hard, battle the many stakeholders all of whom want their thing at the top of the search results page, get a load of extra headcount to make a robust and scalable user-facing system, join an on-call rota and get called at 2am, engage in a bunch of ethically questionable behaviour skirting the border between fair use and copyright infringement, hire and manage loads of data labellers in low-income countries who get paid a pittance, battle the internal doubters who think Google Assistant shows chatbots are a joke and users don't want it, and battle the internal fearmongers who think your ML system is going to call black people monkeys, and at the end of it maybe it's great or maybe it ends up an embarrassment that gets withdrawn, like Tay.

Or you can publish some academic papers. Maybe do some work improving the automatic transcription for youtube, or translation for google translate. Finish work at 3pm on a Friday, and have plenty of time to enjoy your $400k salary.

Re: Google's First Tensor Processing Unit: Architecture

#167

Post by sigmoid10 » Tue, Mar 26, 2024, 3:17 PM UTC

bartwr wrote: post #135

Laremere wrote: post #88

Earlier quoted context omitted.

The way I see, NVidia only has a few advantages ordered from most important to least: 1. Reserved fab space. 2. Highly integrated software. 3. Hardware architecture that exists today. 4. Customer relationships. but all of these aspects are weak in one way or another: For #1, fab space is tight, and NVidia can strangle its consumer GPU market if it means selling more AI chips at a higher price. This advantage is gone…

Seems you have not worked with ML workloads, but base your comment on "internet wisdom", or worse, business analysts (I am sorry if that's inaccurate). On GPUs, ML "just works" (inference and training) and are always order of magnitude faster than whatever CPU you have. TPUs work very well for some model architectures (old ones that they were optimized and designed for) and on some novel others can be actually slower…

>On GPUs, ML "just works"

If you had worked with ML, you'd know that this is not true. It's actually more like the opposite. It also has nothing to do with the chips themselves. Things don't magically work "because GPU", they work because manufacturers spend the time getting their drivers and ecosystems right. That's why for example noone is using AMD GPUs for ML, despite them offering more compute per dollar on paper. Getting the software stack to the point of Nvidia/CUDA, where things really do "just work", is an enormous undertaking. And as someone who has been researching ML for more than a decade now, I can tell you Nvidia also didn't get these things right in the beginning. That's the reason why they have no real competition today (and still won't for quite some time).

Re: Google's First Tensor Processing Unit: Architecture

#168

Post by nemothekid » Tue, Mar 26, 2024, 3:20 PM UTC

varjag wrote: post #164

disgruntledphd2 wrote: post #160

Earlier quoted context omitted.

There's not enough money in it, as Google's scale. Especially because the people who'd pay for Premium tend to be the most prized people from an advertiser perspective. And most people won't pay, under any circumstances, but they will click on ads which make Google money.

YouTube does it, at Google scale. And these same people do pay $20/mo for ChatGPT anyway.

YouTube isn't comparable - YouTube revenue is roughly 30B/year, while Search revenue is roughly 175B/year.

Advertisers are willing to pay far more than $20/mo per user, combined with the fact that search costs way less per query than inference.

Re: Google's First Tensor Processing Unit: Architecture

#169

Post by HarHarVeryFunny » Tue, Mar 26, 2024, 3:28 PM UTC

ipsum2 wrote: post #96

nl wrote: post #29

On the podcast interview now Groq CEO Jonathon Ross did[1] he talked about the creation of the original TPUs (which he built at Google). Apparently originally it was a FPGA he did in his 20% time because he sat near the team who was having inference speed issues. They got it working, then Jeff Dean did the math and the decided to do an ASIC. Now of course Google should spin off the TPU team as a separate company. It'…

> It's the only credible competition NVidia has This is wrong, both AMD and Intel (through Habana) have GPUs comparable to H100s in performance.

There's also Amazon's AWS "Trainium" chips, which is what Anthropic will be using going forward.

If you're talking about training LLMs, involving 10's of thousands of processors, then the specifics of one processor vs another isn't the most important thing - it's the overall architecture and infrastructure in place to manage it.

Re: Google's First Tensor Processing Unit: Architecture

#170

Post by HarHarVeryFunny » Tue, Mar 26, 2024, 3:32 PM UTC

brcmthrowaway wrote: post #52

Broadcom did the TPU

Not the whole design - the core processing part (systolic array - matrix multiplier) was designed by Google, but Broadcom designed all the highspeed chip I/O and mapped the design onto TSMCs tools/rules.