How large are large language models? • Hacker News Board

Re: How large are large language models?

#21

Post by rain1 » Wed, Jul 02, 2025, 12:16 PM UTC

christianqchung wrote: post #17

This is a bad article. Some of the information is wrong, and it's missing lots of context. For example, it somehow merged Llama 4 Maverick's custom Arena chatbot version with Behemoth, falsely claiming that the former is stopping the latter from being released. It also claims 40B of internet text data is 10B tokens, which seems a little odd. Llama 405B was also trained on more than 15 trillion tokens[1], but the post…

I can correct mistakes.

> it somehow merged Llama 4 Maverick's custom Arena chatbot version with Behemoth

I can clarify this part. I wrote 'There was a scandal as facebook decided to mislead people by gaming the lmarena benchmark site - they served one version of llama-4 there and released a different model' which is true.

But it is inside the section about the llama 4 model behemoth. So I see how that could be confusing/misleading.

I could restructure that section a little to improve it.

> Llama 405B was also trained on more than 15 trillion tokens[1],

You're talking about Llama 405B instruct, I'm talking about Llama 405B base. Of course the instruct model has been traiend on more tokens.

> why is there such a focus on token training count?

I tried to include the rough training token count for each model I wrote about - plus additional details about training data mixture if available. Training data is an important part of an LLM.

Re: How large are large language models?

#22

Post by OtherShrezzing » Wed, Jul 02, 2025, 12:30 PM UTC

rain1 wrote: post #15

OtherShrezzing wrote: post #4

>None of this document was not written by AI I think in these scenarios, articles should include the prompt and generating model.

I have corrected that. It was supposed to say "None of this document was written by AI." Thank you for spotting the error.

Understood, thanks for updating it!

Re: How large are large language models?

#23

Post by charcircuit » Wed, Jul 02, 2025, 12:37 PM UTC

mjburgess wrote: post #5

Deepseek v1 is ~670Bn which is ~1.4TB physical. All digitized books ever written/encoded compress to a few TB. The public web is ~50TB. I think a usable zip of all english electronic text publicly available would be on O(100TB). So we're at about 1% of that in model size, and we're in a diminishing-returns area of training -- ie., going to >1% has not yielded improvements (cf. gpt4.5 vs 4o). This is why compute spend…

>The public web is ~50TB

Did you mean to type EB?

Re: How large are large language models?

#24

Post by andrepd » Wed, Jul 02, 2025, 12:46 PM UTC

mjburgess wrote: post #5

Deepseek v1 is ~670Bn which is ~1.4TB physical. All digitized books ever written/encoded compress to a few TB. The public web is ~50TB. I think a usable zip of all english electronic text publicly available would be on O(100TB). So we're at about 1% of that in model size, and we're in a diminishing-returns area of training -- ie., going to >1% has not yielded improvements (cf. gpt4.5 vs 4o). This is why compute spend…

> 50TB

There's no way the entire Web fits in 400$ worth of hard drives.

Re: How large are large language models?

#25

Post by AlienRobot » Wed, Jul 02, 2025, 12:54 PM UTC

andrepd wrote: post #24

mjburgess wrote: post #5

Deepseek v1 is ~670Bn which is ~1.4TB physical. All digitized books ever written/encoded compress to a few TB. The public web is ~50TB. I think a usable zip of all english electronic text publicly available would be on O(100TB). So we're at about 1% of that in model size, and we're in a diminishing-returns area of training -- ie., going to >1% has not yielded improvements (cf. gpt4.5 vs 4o). This is why compute spend…

> 50TB There's no way the entire Web fits in 400$ worth of hard drives.

Text is small.

Re: How large are large language models?

#26

Post by fossa1 » Wed, Jul 02, 2025, 12:56 PM UTC

It’s ironic: for years the open-source community was trying to match GPT-3 (175B dense) with 30B–70B models + RLHF + synthetic data—and the performance gap persisted.

Turns out, size really did matter, at least at the base model level. Only with the release of truly massive dense (405B) or high-activation MoE models (DeepSeek V3, DBRX, etc) did we start seeing GPT-4-level reasoning emerge outside closed labs.

Re: How large are large language models?

#27

Post by fouc » Wed, Jul 02, 2025, 12:57 PM UTC

mjburgess wrote: post #5

Deepseek v1 is ~670Bn which is ~1.4TB physical. All digitized books ever written/encoded compress to a few TB. The public web is ~50TB. I think a usable zip of all english electronic text publicly available would be on O(100TB). So we're at about 1% of that in model size, and we're in a diminishing-returns area of training -- ie., going to >1% has not yielded improvements (cf. gpt4.5 vs 4o). This is why compute spend…

Maybe you're thinking of Library of Congress when you say ~50TB? Internet is definitely larger..

Re: How large are large language models?

#28

Post by flir » Wed, Jul 02, 2025, 12:59 PM UTC

andrepd wrote: post #24

mjburgess wrote: post #5

Deepseek v1 is ~670Bn which is ~1.4TB physical. All digitized books ever written/encoded compress to a few TB. The public web is ~50TB. I think a usable zip of all english electronic text publicly available would be on O(100TB). So we're at about 1% of that in model size, and we're in a diminishing-returns area of training -- ie., going to >1% has not yielded improvements (cf. gpt4.5 vs 4o). This is why compute spend…

> 50TB There's no way the entire Web fits in 400$ worth of hard drives.

Nah, Common Crawl puts on 250TB a month.

Maybe text only, though...

Re: How large are large language models?

#29

Post by stared » Wed, Jul 02, 2025, 1:00 PM UTC

If you want it visually, here's a chart of total parameters as a function of year: https://app.charts.quesma.com/s/rmyk38

Re: How large are large language models?

#30

Post by ljoshua » Wed, Jul 02, 2025, 1:00 PM UTC

Less a technical comment and more just a mind-blown comment, but I still can’t get over just how much data is compressed into and available in these downloadable models. Yesterday I was on a plane with no WiFi, but had gemma3:12b downloaded through Ollama. Was playing around with it and showing my kids, and we fired history questions at it, questions about recent video games, and some animal fact questions. It wasn’t perfect, but holy cow the breadth of information that is embedded in an 8.1 GB file is incredible! Lossy, sure, but a pretty amazing way of compressing all of human knowledge into something incredibly contained.