Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Meta trains large language models at scale

engineering.fb.com

Post reply on HN
Print view View original

191–200 of 213 posts

Re: How Meta trains large language models at scale

#191

Post by koe123 »

refulgentis wrote: post #110
vineyardmike wrote: post #100

Earlier quoted context omitted.

> why wouldn't Google be selling shovels They do sell them - but through their struggling cloud business. Either way, Nvidia's margin is google's opportunity to lower costs. > I can see Google's custom chips are 15x to 30x slower to train AI TPUs are designed for inference not training - they're betting that they can serve models to the world at a lower cost structure than their competition. The compute required for…

This breaks my brain, because I know Google trains it models on TPUs and they're seen as faster, and if they're better at inference, and can train, then why is Nvidia in a unique position? My understanding was always it's as simple as it required esoteric tooling

Possibly naive but I very much view CUDA and its integration into ML frameworks being nvidias moat

Re: How Meta trains large language models at scale

#192

Post by claytonjy »

p4ul wrote: post #182
radarsat1 wrote: post #138

Frustratingly little information. For example, I'm exceedingly curious how they deal with scheduling jobs on such a huge array of machines. The article: > Efficient scheduling helps ensure that our resources are used optimally. This involves sophisticated algorithms that can allocate resources based on the needs of different jobs and dynamic scheduling to adapt to changing workloads. Wow thanks for that, captain obvi…

I usually assume these companies are using some of the popular schedulers (e.g., Slurm, MOAB, SGE) that have existed in the HPC community for many years. I have anecdotally also heard that some are using k8s, but I've not seen that myself. Slurm [1] is basically built for this stuff; that's definitely what I would use! [1] https://slurm.schedmd.com/documentation.html

Slurm is definitely still dominant, but OpenAI has been using k8s for training for many years now¹, and there are various ways to run slurm on top of Kubernetes, including the recent SUNK from coreweave²

at my company we use slurm "directly" for static compute we rent or own (i.e. not in a public cloud), but are considering using Kubernetes because that's how we run the rest of the company, and we'd rather invest more effort into being better at k8s than becoming good slurm admins.

¹: https://openai.com/index/scaling-kubernetes-to-2500-nodes/

²: https://www.coreweave.com/blog/sunk-slurm-on-kubernetes-impl...

Re: How Meta trains large language models at scale

#193

Post by jejeyyy77 »

mike_d wrote: post #4

Posts like this underscore why the smart money is betting on Google as the long term AI winner. Meta, Microsoft, OpenAI, etc. are trying to address problems with consumer video cards and spending billions to try and out bid each other to win Nvidia's favor - while Google is on their 6th generation of custom silicon. Literally the only thing that can stop Google now is the fact they keep bringing Microsoft and Oracle…

lolwat.

google is the biggest loser in all of this.

Re: How Meta trains large language models at scale

#194

Post by p4ul »

claytonjy wrote: post #192
p4ul wrote: post #182

Earlier quoted context omitted.

I usually assume these companies are using some of the popular schedulers (e.g., Slurm, MOAB, SGE) that have existed in the HPC community for many years. I have anecdotally also heard that some are using k8s, but I've not seen that myself. Slurm [1] is basically built for this stuff; that's definitely what I would use! [1] https://slurm.schedmd.com/documentation.html

Slurm is definitely still dominant, but OpenAI has been using k8s for training for many years now¹, and there are various ways to run slurm on top of Kubernetes, including the recent SUNK from coreweave² at my company we use slurm "directly" for static compute we rent or own (i.e. not in a public cloud), but are considering using Kubernetes because that's how we run the rest of the company, and we'd rather invest mor…

Very cool! Thank for this, claytonjy!!

Re: How Meta trains large language models at scale

#195

Post by _zoltan_ »

idkdotcom wrote: post #54

These seem classic challenges with running distributed systems loads that are not specific to training LLMs. Anyone of the super computers listed here https://en.wikipedia.org/wiki/TOP500 suffers from the same issues. Think about it. While the national labs use these systems to model serious stuff -such as climate or nuclear weapons- Meta uses them to train LLMs. What a joke, honestly!

For profit is not less serious than what research labs do. I'd even say it's more important: they drive the economy.

Re: How Meta trains large language models at scale

#196

Post by idkdotcom »

mhandley wrote: post #166
idkdotcom wrote: post #54

These seem classic challenges with running distributed systems loads that are not specific to training LLMs. Anyone of the super computers listed here https://en.wikipedia.org/wiki/TOP500 suffers from the same issues. Think about it. While the national labs use these systems to model serious stuff -such as climate or nuclear weapons- Meta uses them to train LLMs. What a joke, honestly!

On the other hand, Meta just rapidly built two different training networks in existing datacenter buildings, with existing cooling constraints, using mostly commodity components (albeit expensive commodity components) each of which would place at #3 on that top500 list in terms of GPU power. Compare that with how long it took to get any of the other supercomputers from design to being fully commissioned.

[dead]

Re: How Meta trains large language models at scale

#197

Post by refulgentis »

vineyardmike wrote: post #185
refulgentis wrote: post #110

Earlier quoted context omitted.

This breaks my brain, because I know Google trains it models on TPUs and they're seen as faster, and if they're better at inference, and can train, then why is Nvidia in a unique position? My understanding was always it's as simple as it required esoteric tooling

Because people generally don’t use TPUs outside of Google. The tooling is different, the access is metered through GCP, etc. Nvidia is in a vaguely unique position in that their products have great tooling support and few companies sell silicon at their scale.

Correct, I'm pointing out politely that's in conflict with the person I'm replying to.

Re: How Meta trains large language models at scale

#198

Post by rmbyrro »

TeMPOraL wrote: post #120
yosito wrote: post #67

I wish that instead of just training another stupid LLM, Meta would use it to improve their search and help me find the content I'm actually interested in.

Their revenue depends on it being hard (but not impossible) for you to find the content you're actually interested in. Would be nice if it didn't, but in this reality, money on the Internet is made by wasting users' lives. That is what attention economy is about.

It's actually a mix. They need to disappoint the user for the right amount of time, to then please it at the right moment and dose. This maximizes the dopamine release and increases addictiveness.

When you find good content depends on when the algo judges you're already primed to a colorful dopamine intake.

Re: How Meta trains large language models at scale

#199

Post by mike_d »

kkielhofner wrote: post #149
mike_d wrote: post #116

Earlier quoted context omitted.

They are building racks of 4090s. Nobody can get H100s in any reasonable volume. Hell, Microsoft is renting GPUs from Oracle Cloud to get enough capacity to run Bing.

Who is "they"? RTX 4090s are terrible for this task. Off the top of my head: - VRAM (obviously). Isn't that where the racks come in? Not really. Nvidia famously removed something as basic as NVLink between two cards from the 3090 to the 4090. When it comes to bandwidth between cards (crucial) even 16 lanes of PCIe 4 isn't fast enough. When you start talking about "racks" unless you're running on server grade CPUs (co…

> but deploying "racks" of 4090s for AI tasks is (frankly) flat-out stupid.

You seem to be trapped in the delusion that this was anyone's first, second, or third choice.

There is workload demand, you can't get H100s, and if you don't start racking up the cards you can get the company will replace you with someone less opinionated.

Re: How Meta trains large language models at scale

#200

Post by nickpsecurity »

throwaway_ab wrote: post #15
mike_d wrote: post #4

Posts like this underscore why the smart money is betting on Google as the long term AI winner. Meta, Microsoft, OpenAI, etc. are trying to address problems with consumer video cards and spending billions to try and out bid each other to win Nvidia's favor - while Google is on their 6th generation of custom silicon. Literally the only thing that can stop Google now is the fact they keep bringing Microsoft and Oracle…

I think it's likely Nvidia's GPU's, many of which are $50,000+ for a single unit, far surpass Google's custom silicon otherwise why wouldn't Google be selling shovels like Nvidia? If Google had a better chip, or even a chip that was close, they would sell it to anyone and everyone. From a quick search I can see Google's custom chips are 15x to 30x slower to train AI compared to Nvidia's current latest gen AI specific…

That’s not necessarily true. Many companies make chips they won’t sell to support lucrative, proprietary offerings. Mainframe processors are the classic example.

In AI, Google (TPU) and Intel (Gaudi) each have chips they push in cloud offerings. The cloud offerings have cross selling opportunities. That by itself would be a reason to keep it internal at their scale. It might also be easier to support one, or a small set, of deployments that are internal vs the variety that external customers would use.

Post reply on HN