Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Meta trains large language models at scale

engineering.fb.com

Post reply on HN
Print view View original

201–210 of 213 posts

Re: How Meta trains large language models at scale

#201

Post by sonofaragorn »

OsrsNeedsf2P wrote: post #124
Oras wrote: post #84

Would be nice to read how do they collect/prepare data for training. Which data sources? How much of Meta users data (fb, instagram… etc). How do they sanitize PII?

> How do they sanitize PII? I can't comment on how things like faces get used, but in my experience, PII at Meta is inaccessible by default. Unless you're impersonating a user on the platform (to access what PII they can see), you have to request special access for logs or database columns that contain so much as user IDs, otherwise the data simply won't show up when you query for it. This is baked into the infrastru…

What about a post or comment that includes proper names?

Re: How Meta trains large language models at scale

#202

Post by dudus »

afro88 wrote: post #168
lannisterstark wrote: post #126

Earlier quoted context omitted.

Well, yes, but Not all models need to be "super large." Smaller models, specialized in specific tasks, working together - and then reporting to a slightly larger model is the way to go. Think of everything being connected to a "Home Computer" in those "Future House of 2020" videos that were out there in 70s or what not. Another example (very rough) would be something like "Weather data gets to a small model via an AP…

> Well, yes, but Not all models need to be "super large." Smaller models, specialized in specific tasks, working together - and then reporting to a slightly larger model is the way to go. I want to believe, but I'm still yet to see this kind of set up being anywhere near GPT-4 level. The weather example seems quite contrived. Why not just display the alerts for your area? Why is a complex system of smaller models rep…

Because cost and speed. Smaller models can run on your phone for free, or on the cloud for pennies. An API call for a large LLM with a lot of context can cost orders of magnitude more and incur network latency

Re: How Meta trains large language models at scale

#203

Post by matt-p »

UCBdaPatterson wrote: post #177
matt-p wrote: post #28

Earlier quoted context omitted.

Exactly and they are still about 1/18ths as good at training llms as a H100. Maybe they are less than 1/18ths the cost, so google technically have a marginally better unit cost but i doubt it when you consider the R&D cost. They are less bad at inference, but still much worse than even an A100.

If you're interested in a peer reviewed scientific comparison, Google writes retrospective papers after contemporary TPUs and GPUs are deployed versus speculation about future products. The most recent compares TPU v4 and A100. (TPU v5 and H100 is for a future paper). Here is a quote from the abstract: "Deployed since 2020, TPU v4 outperforms TPU v3 by 2.1x and improves performance/Watt by 2.7x. ... For similar sized…

I'm sure it probably is faster for thier own workloads (which they are choosing to benchmark on), why bother making it if not. But that is clearly not universally true, a GPU is clearly more versatile. This means nothing to most if they can't for example train an LLM on them.

Re: How Meta trains large language models at scale

#204

Post by rldjbpin »

yosito wrote: post #67

I wish that instead of just training another stupid LLM, Meta would use it to improve their search and help me find the content I'm actually interested in.

this is like asking disney to reduce wait times for their rides. in other words, it is against content aggregation platforms' interest to let you get what you want directly.

Re: How Meta trains large language models at scale

#205

Post by rldjbpin »

mike_d wrote: post #4

Posts like this underscore why the smart money is betting on Google as the long term AI winner. Meta, Microsoft, OpenAI, etc. are trying to address problems with consumer video cards and spending billions to try and out bid each other to win Nvidia's favor - while Google is on their 6th generation of custom silicon. Literally the only thing that can stop Google now is the fact they keep bringing Microsoft and Oracle…

being first/early doesn't always mean long-term dominance, see tensorflow.

at consumer level, npu are becoming a useful accelerator, and here google can choose to become the platform of choice.

but nobody comes close for training workflow but nvidia as it stands. imho it is currently possible thanks to community efforts afforded by cuda being the only realistic option.

politics and leadership aside, it would be nice to sustian a market for highly efficient matrix multipliers. also a s/w ecosystem that finally makes multiprocessing workloads easy to integrate for dummies like me.

Re: How Meta trains large language models at scale

#206

Post by [deleted] »

dudus wrote: post #71

> Since we did not have time to change the cooling infrastructure, we had to remain in an air-cooled environment. The mechanical and thermal designs had to change to accommodate this, and that triggered a validation cycle to support a large-scale deployment. > All of these hardware-related changes were challenging because we had to find a solution that fit within the existing resource constraints, with a very small d…

[deleted]

Re: How Meta trains large language models at scale

#207

Post by nomel »

dudus wrote: post #71

> Since we did not have time to change the cooling infrastructure, we had to remain in an air-cooled environment. The mechanical and thermal designs had to change to accommodate this, and that triggered a validation cycle to support a large-scale deployment. > All of these hardware-related changes were challenging because we had to find a solution that fit within the existing resource constraints, with a very small d…

What's the cost of letting that hardware sit idle?

Re: How Meta trains large language models at scale

#208

Post by nomel »

coder543 wrote: post #180
UCBdaPatterson wrote: post #177

Earlier quoted context omitted.

If you're interested in a peer reviewed scientific comparison, Google writes retrospective papers after contemporary TPUs and GPUs are deployed versus speculation about future products. The most recent compares TPU v4 and A100. (TPU v5 and H100 is for a future paper). Here is a quote from the abstract: "Deployed since 2020, TPU v4 outperforms TPU v3 by 2.1x and improves performance/Watt by 2.7x. ... For similar sized…

That quote is referring to the A100... the H100 used ~75% more power to deliver "up to 9x faster AI training and up to 30x faster AI inference speedups on large language models compared to the prior generation A100."[0] Which sure makes the H100 sound both faster and more efficient (per unit of compute) than the TPU v4, given what was in your quote. I don't think your quote does anything to support the position that…

As someone unfamiliar with this area, can one of the downvotes explain why they choose to downvote this? Is it wrong?

Re: How Meta trains large language models at scale

#209

Post by mhsiu »

bluedino wrote: post #59
throwaway_ab wrote: post #15

Earlier quoted context omitted.

I think it's likely Nvidia's GPU's, many of which are $50,000+ for a single unit, far surpass Google's custom silicon otherwise why wouldn't Google be selling shovels like Nvidia? If Google had a better chip, or even a chip that was close, they would sell it to anyone and everyone. From a quick search I can see Google's custom chips are 15x to 30x slower to train AI compared to Nvidia's current latest gen AI specific…

We have almost 400 H100's sitting idle. I wonder how many other companies are buying millions of dollars worth of these chips with the hopes of them being used, but aren't being utilized?

If you'd like to contribute some much-appreciated compute to university researchers all across the US, please email me (mhsiu at ucsd.edu).

It should be possible to hook up your idle devices to Nautilus (https://nationalresearchplatform.org/), which is a cluster set up to support researchers at a bunch of universities. I can't guarantee anything since I'm not involved in the cluster management itself, but I can put you in contact with those who are if you're interested.

Re: How Meta trains large language models at scale

#210

Post by idkdotcom »

_zoltan_ wrote: post #195
idkdotcom wrote: post #54

These seem classic challenges with running distributed systems loads that are not specific to training LLMs. Anyone of the super computers listed here https://en.wikipedia.org/wiki/TOP500 suffers from the same issues. Think about it. While the national labs use these systems to model serious stuff -such as climate or nuclear weapons- Meta uses them to train LLMs. What a joke, honestly!

For profit is not less serious than what research labs do. I'd even say it's more important: they drive the economy.

[dead]
Post reply on HN