Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How large are large language models?

gist.github.com

Post reply on HN
Print view View original

41–50 of 160 posts

Re: How large are large language models?

#41

Post by exe34 »

ljoshua wrote: post #30

Less a technical comment and more just a mind-blown comment, but I still can’t get over just how much data is compressed into and available in these downloadable models. Yesterday I was on a plane with no WiFi, but had gemma3:12b downloaded through Ollama. Was playing around with it and showing my kids, and we fired history questions at it, questions about recent video games, and some animal fact questions. It wasn’t…

Wikipedia is about 24GB, so if you're allowed to drop 1/3 of the details and make up the missing parts by splicing in random text, 8GB doesn't sound too bad.

To me the amazing thing is that you can tell the model to do something, even follow simple instructions in plain English, like make a list or write some python code to do $x, that's the really amazing part.

Re: How large are large language models?

#42

Post by stared »

rain1 wrote: post #31
stared wrote: post #29

If you want it visually, here's a chart of total parameters as a function of year: https://app.charts.quesma.com/s/rmyk38

This is really awesome. Thank you for creating that. I included a screenshot and link to the chart with credit to you in a comment to my post.

I am happy you like it!

If you like darker color scheme, here it is:

https://app.charts.quesma.com/s/f07qji

And active vs total:

https://app.charts.quesma.com/s/4bsqjs

Re: How large are large language models?

#43

Post by b112 »

exe34 wrote: post #41
ljoshua wrote: post #30

Less a technical comment and more just a mind-blown comment, but I still can’t get over just how much data is compressed into and available in these downloadable models. Yesterday I was on a plane with no WiFi, but had gemma3:12b downloaded through Ollama. Was playing around with it and showing my kids, and we fired history questions at it, questions about recent video games, and some animal fact questions. It wasn’t…

Wikipedia is about 24GB, so if you're allowed to drop 1/3 of the details and make up the missing parts by splicing in random text, 8GB doesn't sound too bad. To me the amazing thing is that you can tell the model to do something, even follow simple instructions in plain English, like make a list or write some python code to do $x, that's the really amazing part.

Not to mention, Language Modeling is Compression https://arxiv.org/pdf/2309.10668

So text wikipedia at 24G would easily hit 8G with many standard forms of compression, I'd think. If not better. And it would be 100% accurate, full text and data. Far more usable.

It's so easy for people to not realise how massive 8GB really is, in terms of text. Especially if you use ascii instead of UTF.

Re: How large are large language models?

#44

Post by MPSimmons »

rain1 wrote: post #36
ljoshua wrote: post #30

Less a technical comment and more just a mind-blown comment, but I still can’t get over just how much data is compressed into and available in these downloadable models. Yesterday I was on a plane with no WiFi, but had gemma3:12b downloaded through Ollama. Was playing around with it and showing my kids, and we fired history questions at it, questions about recent video games, and some animal fact questions. It wasn’t…

It's extremely interesting how powerful a language model is at compression. When you train it to be an assistant model, it's better at compressing assistant transcripts than it is general text. There is an eval which I have a lot of interested in and respect for https://huggingface.co/spaces/Jellyfish042/UncheatableEval called UncheatableEval, which tests how good of a language model an LLM is by applying it on a ran…

Agreed. It's basically lossy compression for everything it's ever read. And the quantization impacts the lossiness, but since a lot of text is super fluffy, we tend not to notice as much as we would when we, say, listen to music that has been compressed in a lossy way.

Re: How large are large language models?

#45

Post by rybosome »

simonw wrote: post #38

> There were projects to try to match it, but generally they operated by fine tuning things like small (70B) llama models on a bunch of GPT-3 generated texts (synthetic data - which can result in degeneration when AI outputs are fed back into AI training inputs). That parenthetical doesn't quite work for me. If synthetic data always degraded performance, AI labs wouldn't use synthetic data. They use it because it hel…

Agreed, especially when in this context of training a smaller model on a larger model’s outputs. Distillation is generally accepted as an effective technique.

This is exactly what I did in a previous role, fine-tuning Llama and Mistral models on a mix of human and GPT-4 data for a domain-specific task. Adding (good) synthetic data definitely increased the output quality for our tasks.

Re: How large are large language models?

#46

Post by horsawlarway »

b112 wrote: post #43
exe34 wrote: post #41

Earlier quoted context omitted.

Wikipedia is about 24GB, so if you're allowed to drop 1/3 of the details and make up the missing parts by splicing in random text, 8GB doesn't sound too bad. To me the amazing thing is that you can tell the model to do something, even follow simple instructions in plain English, like make a list or write some python code to do $x, that's the really amazing part.

Not to mention, Language Modeling is Compression https://arxiv.org/pdf/2309.10668 So text wikipedia at 24G would easily hit 8G with many standard forms of compression, I'd think. If not better. And it would be 100% accurate, full text and data. Far more usable. It's so easy for people to not realise how massive 8GB really is, in terms of text. Especially if you use ascii instead of UTF.

The 24G is the compressed number.

They host a pretty decent article here: https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia

The relevant bit:

> As of 16 October 2024, the size of the current version including all articles compressed is about 24.05 GB without media.

Re: How large are large language models?

#47

Post by phkahler »

ljlolel wrote: post #34
ljoshua wrote: post #30

Less a technical comment and more just a mind-blown comment, but I still can’t get over just how much data is compressed into and available in these downloadable models. Yesterday I was on a plane with no WiFi, but had gemma3:12b downloaded through Ollama. Was playing around with it and showing my kids, and we fired history questions at it, questions about recent video games, and some animal fact questions. It wasn’t…

How big is Wikipedia text? Within 3X that size with 100% accuracy

Google AI response says this for compressed size of wikipedia:

"The English Wikipedia, when compressed, currently occupies approximately 24 GB of storage space without media files. This compressed size represents the current revisions of all articles, but excludes media files and previous revisions of pages, according to Wikipedia and Quora."

So 3x is correct but LLMs are lossy compression.

Re: How large are large language models?

#48

Post by 1vuio0pswjnm7 »

1. "raw text continuation engine"

https://gist.github.com/rain-1/cf0419958250d15893d8873682492...

2. "superintelligence"

https://en.m.wikipedia.org/wiki/Superintelligence

"Meta is uniquely positioned to deliver superintelligence to the world."

https://www.cnbc.com/2025/06/30/mark-zuckerberg-creating-met...

Is there any difference between 1 and 2

Yes. One is purely hypothetical

Re: How large are large language models?

#49

Post by Workaccount2 »

ljoshua wrote: post #30

Less a technical comment and more just a mind-blown comment, but I still can’t get over just how much data is compressed into and available in these downloadable models. Yesterday I was on a plane with no WiFi, but had gemma3:12b downloaded through Ollama. Was playing around with it and showing my kids, and we fired history questions at it, questions about recent video games, and some animal fact questions. It wasn’t…

I don't like the term "compression" used with transformers because it gives the wrong idea about how they function. Like that they are a search tool glued onto a .zip file, your prompts are just fancy search queries, and hallucinations are just bugs in the recall algo.

Although strictly speaking they have lots of information in a small package, they are F-tier compression algorithms because the loss is bad, unpredictable, and undetectable (i.e. a human has to check it). You would almost never use a transformer in place of any other compression algorithm for typical data compression uses.

Re: How large are large language models?

#50

Post by ben_w »

TeMPOraL wrote: post #20
account-5 wrote: post #8

Earlier quoted context omitted.

> All digitized books ever written/encoded compress to a few TB. The public web is ~50TB. I think a usable zip of all english electronic text publicly available would be on O(100TB). Where you getting these numbers from? Interested to see how that's calculated. I read somewhere, but cannot find the source anymore, that all written text prior to this century was approx 50MB. (Might be misquoted as don't have source an…

> I read somewhere, but cannot find the source anymore, that all written text prior to this century was approx 50MB. (Might be misquoted as don't have source anymore). 50 MB feels too low, unless the quote meant text up until the 20th century , in which case it feels much more believable. In terms of text production and publishing, we're still riding an exponent, so a couple orders of magnitude increase between 1899…

Even by the start of the 20th century, 50 MB is definitely far too low.

Any given English translation of Bible is by itself something like 3-5 megabytes of ASCII; the complete works of Shakespeare are about 5 megabytes; and I think (back of the envelope estimate) you'd get about the same again for what Arthur Conan Doyle wrote before 1900.

I can just about believe there might have been only ten thousand Bible-or-Shakespeare sized books (plus all the court documents, newspapers, etc. that add up to that) worldwide by 1900, but not ten.

Edit: I forgot about encyclopaedias, by 1900 the Encyclopædia Britannica was almost certainly more than 50 MB all by itself.

Post reply on HN