How Meta trains large language models at scale • Hacker News Board

Re: How Meta trains large language models at scale

#81

Post by asynchronous » Thu, Jun 13, 2024, 3:24 AM UTC

mike_d wrote: post #4

Posts like this underscore why the smart money is betting on Google as the long term AI winner. Meta, Microsoft, OpenAI, etc. are trying to address problems with consumer video cards and spending billions to try and out bid each other to win Nvidia's favor - while Google is on their 6th generation of custom silicon. Literally the only thing that can stop Google now is the fact they keep bringing Microsoft and Oracle…

Laughable response when you actually look at the quality of the algorithms being produced by Google. They’re so behind it’s embarrassing.

Re: How Meta trains large language models at scale

#82

Post by rapsey » Thu, Jun 13, 2024, 3:28 AM UTC

candiddevmike wrote: post #21

dinobones wrote: post #10

Earlier quoted context omitted.

Do you really think Google’s hardware expertise is better than Nvidia’s? If needed these other companies have the $$$ to buy the best chips money can buy from Nvidia. Better chips than Google could ever produce. If anything, this is why IMO Google will fail.

I thought NVIDIA's moat was mostly software/CUDA?

They have by far the fastest chip, the best software with CUDA and feverishly working on next gen chips. They also bought out multiple years worth of global high bandwidth memory manufacturing capacity.

No one will beat them at their game. However if there are any major breakthroughs that might render those processing capacities unneeded, or the major players hitting a wall regarding AI spending, then they will take a massive hit. It will come eventually because the chip business is always in boom/bust cycles.

Re: How Meta trains large language models at scale

#83

Post by giancarlostoro » Thu, Jun 13, 2024, 3:29 AM UTC

bluedino wrote: post #59

throwaway_ab wrote: post #15

Earlier quoted context omitted.

I think it's likely Nvidia's GPU's, many of which are $50,000+ for a single unit, far surpass Google's custom silicon otherwise why wouldn't Google be selling shovels like Nvidia? If Google had a better chip, or even a chip that was close, they would sell it to anyone and everyone. From a quick search I can see Google's custom chips are 15x to 30x slower to train AI compared to Nvidia's current latest gen AI specific…

We have almost 400 H100's sitting idle. I wonder how many other companies are buying millions of dollars worth of these chips with the hopes of them being used, but aren't being utilized?

Probably could profit selling them second hand honestly.

Re: How Meta trains large language models at scale

#84

Post by Oras » Thu, Jun 13, 2024, 3:30 AM UTC

Would be nice to read how do they collect/prepare data for training.

Which data sources? How much of Meta users data (fb, instagram… etc). How do they sanitize PII?

Re: How Meta trains large language models at scale

#85

Post by raincole » Thu, Jun 13, 2024, 3:33 AM UTC

mike_d wrote: post #4

Posts like this underscore why the smart money is betting on Google as the long term AI winner. Meta, Microsoft, OpenAI, etc. are trying to address problems with consumer video cards and spending billions to try and out bid each other to win Nvidia's favor - while Google is on their 6th generation of custom silicon. Literally the only thing that can stop Google now is the fact they keep bringing Microsoft and Oracle…

^ How to compact as many mistakes as possible in one single comment.

1. Google's stock didn't siginificantly outperformed Meta, Microsoft, etc, in thet past two years.

2. Meta and Microsoft are trying to make their own chips as well.

3. They're not using "consumer video cards" to train AI. I don't even know if you can call these beasts video cards any more. H100 doesn't have HDMI port.

Re: How Meta trains large language models at scale

#86

Post by derefr » Thu, Jun 13, 2024, 3:34 AM UTC

matt-p wrote: post #28

candiddevmike wrote: post #19

Earlier quoted context omitted.

They do sell shovels, you can get Google TPUs on Google Cloud.

Exactly and they are still about 1/18ths as good at training llms as a H100. Maybe they are less than 1/18ths the cost, so google technically have a marginally better unit cost but i doubt it when you consider the R&D cost. They are less bad at inference, but still much worse than even an A100.

Given that Google invented Transformer architecture (and Google AI continues to do foundational R&D on ML architecture) — and that Google's TPUs don't even support the most common ML standards, but require their own training and inference frameworks — I would assume that "the point" of TPUs from Google's perspective, has less to do with running LLMs, and more to do with running weird experimental custom model architectures that don't even exist as journal papers yet.

I would bet money that TPUs are at least better at doing AI research than anything Nvidia will sell you. That alone might be enough for Google to keep getting some new ones fabbed each year. The TPUs you can rent on Google Cloud might very well just be hardware requisitioned by the AI team, for the AI team, that they aren't always using to capacity, and so is "earning out" its CapEx through public rentals.

TPUs are maybe also better at other things Google does internally, too. Running inference on YouTube's audio+video-input timecoded-captions-output model, say.

Re: How Meta trains large language models at scale

#87

Post by yellow_postit » Thu, Jun 13, 2024, 3:39 AM UTC

loeg wrote: post #16

mike_d wrote: post #4

Posts like this underscore why the smart money is betting on Google as the long term AI winner. Meta, Microsoft, OpenAI, etc. are trying to address problems with consumer video cards and spending billions to try and out bid each other to win Nvidia's favor - while Google is on their 6th generation of custom silicon. Literally the only thing that can stop Google now is the fact they keep bringing Microsoft and Oracle…

The only thing that can stop Google is Google. Somehow every bet that isn't Search doesn't pan out. And inexplicably, they're working hard to kill Search now. As a shareholder, I hope they succeed. But I am more pessimistic about it than you.

And they missed multiple waves of effectively building on their own in house research.

Re: How Meta trains large language models at scale

#88

Post by yellow_postit » Thu, Jun 13, 2024, 3:43 AM UTC

jauer wrote: post #33

brainfog wrote: post #12

Earlier quoted context omitted.

Because it's completely irrelevant.

and deceptive if not inaccurate. Meta's Model Cards specifically call out that they were trained on publicly available datasets and NOT any Meta user data. For example: https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md

This model card seems at odds with what they claim you can and cannot opt out of [1]

> We use information that is publicly available online and licensed information. We also use information shared on Meta’s Products and services. This information could be things like posts or photos and their captions.

[1] https://www.facebook.com/privacy/genai/?_rdr

Re: How Meta trains large language models at scale

#89

Post by [deleted] » Thu, Jun 13, 2024, 3:45 AM UTC

mike_d wrote: post #4

Posts like this underscore why the smart money is betting on Google as the long term AI winner. Meta, Microsoft, OpenAI, etc. are trying to address problems with consumer video cards and spending billions to try and out bid each other to win Nvidia's favor - while Google is on their 6th generation of custom silicon. Literally the only thing that can stop Google now is the fact they keep bringing Microsoft and Oracle…

[deleted]

Re: How Meta trains large language models at scale

#90

Post by radq » Thu, Jun 13, 2024, 3:47 AM UTC

bluedino wrote: post #59

throwaway_ab wrote: post #15

Earlier quoted context omitted.

I think it's likely Nvidia's GPU's, many of which are $50,000+ for a single unit, far surpass Google's custom silicon otherwise why wouldn't Google be selling shovels like Nvidia? If Google had a better chip, or even a chip that was close, they would sell it to anyone and everyone. From a quick search I can see Google's custom chips are 15x to 30x slower to train AI compared to Nvidia's current latest gen AI specific…

We have almost 400 H100's sitting idle. I wonder how many other companies are buying millions of dollars worth of these chips with the hopes of them being used, but aren't being utilized?

Have you considered sponsoring an open-source project? ;)