How Meta trains large language models at scale • Hacker News Board

Re: How Meta trains large language models at scale

#211

Post by takumif » Fri, Jun 14, 2024, 6:34 AM UTC

123yawaworht456 wrote: post #161

kdot wrote: post #45

How will Meta leverage LLMs at scale to drive revenue? It's not clear.

1. improving their adtech. someone else's API offerings are not an option due to the sheer volume, PII and whatnot. 2. virtually free moderation for their existing (facebook, instagram, threads) and future social media services. likewise, their volume is too insane to even consider paying someone else to process it. the models they do release are probably toys in comparison to their internal models.

> the models they do release are probably toys in comparison to their internal models.

I'm not sure if that's true. They did say they're going to make Llama 3 400B public and if they follow through, that's no toy and I doubt they have something significantly better internally.

Re: How Meta trains large language models at scale

#212

Post by Muhtasham » Sat, Jun 15, 2024, 12:57 AM UTC

nomel wrote: post #207

dudus wrote: post #71

> Since we did not have time to change the cooling infrastructure, we had to remain in an air-cooled environment. The mechanical and thermal designs had to change to accommodate this, and that triggered a validation cycle to support a large-scale deployment. > All of these hardware-related changes were challenging because we had to find a solution that fit within the existing resource constraints, with a very small d…

What's the cost of letting that hardware sit idle?

it’s serious crime

Re: How Meta trains large language models at scale

#213

Post by lannisterstark » Sun, Jun 16, 2024, 12:14 AM UTC

afro88 wrote: post #168

lannisterstark wrote: post #126

Earlier quoted context omitted.

Well, yes, but Not all models need to be "super large." Smaller models, specialized in specific tasks, working together - and then reporting to a slightly larger model is the way to go. Think of everything being connected to a "Home Computer" in those "Future House of 2020" videos that were out there in 70s or what not. Another example (very rough) would be something like "Weather data gets to a small model via an AP…

> Well, yes, but Not all models need to be "super large." Smaller models, specialized in specific tasks, working together - and then reporting to a slightly larger model is the way to go. I want to believe, but I'm still yet to see this kind of set up being anywhere near GPT-4 level. The weather example seems quite contrived. Why not just display the alerts for your area? Why is a complex system of smaller models rep…

>Why not just display the alerts for your area?

Because "Flood warning on roads" is very different than "hey there is a possible tornado/aftershocks/*" and I'd like the 2nd one to take up the entire home-assistant dashboard.

I can either code it myself, or let the model figure it out after passing it the YAML file for the dashboard.

>Why is a complex system of smaller models reporting up to a slightly larger model necessary?

because as the other poster said, cost and speed. thousands of queries every day (potentially) at 30 cents per million tokens is very different than 15 dollars per million tokens.