Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How large are large language models?

gist.github.com

Post reply on HN
Print view View original

131–140 of 160 posts

Re: How large are large language models?

#131

Post by inopinatus »

sharkjacobs wrote: post #80
nico wrote: post #67

Earlier quoted context omitted.

For reference (according to Google): > The English Wikipedia, as of June 26, 2025, contains over 7 million articles and 63 million pages. The text content alone is approximately 156 GB, according to Wikipedia's statistics page. When including all revisions, the total size of the database is roughly 26 terabytes (26,455 GB)

better point of reference might be pages-articles-multistream.xml.bz2 (current pages without edit/revision history, no talk pages, no user pages) which is 20GB https://en.wikipedia.org/wiki/Wikipedia:Database_download#Wh... ?

this is a much more deserving and reliable candidate for any labels regarding the breadth of human knowledge.

Re: How large are large language models?

#132

Post by agumonkey »

antisthenes wrote: post #127
hamilyon2 wrote: post #124

Earlier quoted context omitted.

Crystallized intelligence is. I am not sure about fluid intelligence.

Fluid intelligence is just how quickly you acquire crystallized intelligence. It's the first derivative.

Talking about that, people designed a memory game, dual n back, which allegedly improve fluid intelligence.

Re: How large are large language models?

#133

Post by inopinatus »

mr_toad wrote: post #121
ljoshua wrote: post #30

Less a technical comment and more just a mind-blown comment, but I still can’t get over just how much data is compressed into and available in these downloadable models. Yesterday I was on a plane with no WiFi, but had gemma3:12b downloaded through Ollama. Was playing around with it and showing my kids, and we fired history questions at it, questions about recent video games, and some animal fact questions. It wasn’t…

I will never tire of pointing out that machine learning models are compression algorithms, not compressed data.

I kinda made an argument the other day that they are high-dimensional lossy decompression algorithms, which might be the same difference but looking the other way through the lens.

Re: How large are large language models?

#134

Post by crazygringo »

pessimizer wrote: post #123
pcrh wrote: post #119

Earlier quoted context omitted.

Wikipedia itself describes its size as ~25GB without media [0]. And it's probably more accurate and with broader coverage in multiple languages compared to the LLM downloaded by the GP. https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia

Really? I'd assume that an LLM would deduplicate Wikipedia into something much smaller than 25GB. That's its only job.

> That's its only job.

The vast, vast majority of LLM knowledge is not found in Wikipedia. It is definitely not its only job.

Re: How large are large language models?

#135

Post by Lu2025 »

dgrabla wrote: post #58
ljoshua wrote: post #30

Less a technical comment and more just a mind-blown comment, but I still can’t get over just how much data is compressed into and available in these downloadable models. Yesterday I was on a plane with no WiFi, but had gemma3:12b downloaded through Ollama. Was playing around with it and showing my kids, and we fired history questions at it, questions about recent video games, and some animal fact questions. It wasn’t…

Back in the '90s, we joked about putting “the internet” on a floppy disk. It’s kind of possible now.

Yeah, those guys managed to steal the internet.

Re: How large are large language models?

#136

Post by Aachen »

fouc wrote: post #27
mjburgess wrote: post #5

Deepseek v1 is ~670Bn which is ~1.4TB physical. All digitized books ever written/encoded compress to a few TB. The public web is ~50TB. I think a usable zip of all english electronic text publicly available would be on O(100TB). So we're at about 1% of that in model size, and we're in a diminishing-returns area of training -- ie., going to >1% has not yielded improvements (cf. gpt4.5 vs 4o). This is why compute spend…

Maybe you're thinking of Library of Congress when you say ~50TB? Internet is definitely larger..

Indeed, a quick lookup doesn't give many reliable-sounding sources but they're all on the order of zettabytes (tens to thousands of them), also for years before any LLM was halfway usable. One has to wonder how much of that is generated, thinking of point of my own websites where the pages are derived statistics from player highscores, or the websites that jokingly index all Bitcoin addresses and UUIDs

Perhaps the 50TB estimate is unique information without any media or so, but OP can back up where they got that number from than I can do with guesswork

Re: How large are large language models?

#137

Post by all2 »

oceanplexian wrote: post #129
Wowfunhappy wrote: post #104

Earlier quoted context omitted.

> Common belief, but false. You start learning from inside the womb. The data flow increases exponentially when you open your eyes and then again when you start manipulating things with your hands and mouth. But you don't get data equal to the entire internet as a child! > We do the same with children. At least I did it to my classmates when they asked me for help. I'd give them a hint, and ask them to work it out st…

Your field of vision is equivalent to something like 500 Megapixels. And assume it’s uncompressed because it’s not like your eyeballs are doing H.264. Given vision and the other senses, I’d argue that your average toddler has probably trained on more sensory information than the largest LLMs ever built long before they learn to talk.

There's an adaptation in there somewhere, though. Humans have a 'field of view' that constrains input data, and on the data processing side we have a 'center of focus' that generally rests wherever the eye rests (there's an additional layer where people learn to 'search' their vision by moving their mental center of focus without moving the physical focus point of the eye.

Then there's the whole slew of processes that pick up two or three key points of data and then fill in the rest (EX the moonwalking bear experiment [0]).

I guess all I'm saying is that raw input isn't the only piece of the puzzle. Maybe it is at the start before a kiddo _knows_ how to focus and filter info?

[0] https://www.youtube.com/watch?v=xNSgmm9FX2s

Re: How large are large language models?

#138

Post by ToValueFunfetti »

goatlover wrote: post #108
ToValueFunfetti wrote: post #90

Earlier quoted context omitted.

Under the predictive coding model (and I'm sure some others), animal intelligence is also compression. The idea is that the early layers of the brain minimize how surprising incoming sensory signals are, so the later layers only have to work with truly entropic signal. But it has non-compression-based intelligence within those more abstract layers.

I just wonder if neuroscientists use that kind of model.

I doubt there's any consensus on one model, but it's certainly true that many neuroscientists are using the predictive coding at least some of the time

https://scholar.google.com/scholar?hl=en&as_sdt=0%2C36&q=pre...

Re: How large are large language models?

#139

Post by camel-cdr »

mjburgess wrote: post #5

Deepseek v1 is ~670Bn which is ~1.4TB physical. All digitized books ever written/encoded compress to a few TB. The public web is ~50TB. I think a usable zip of all english electronic text publicly available would be on O(100TB). So we're at about 1% of that in model size, and we're in a diminishing-returns area of training -- ie., going to >1% has not yielded improvements (cf. gpt4.5 vs 4o). This is why compute spend…

> All digitized books ever written/encoded compress to a few TB.

I tied to estimate how much data this actually is:

    # annas archive stats
    papers = 105714890
    books = 52670695
    
    # word count estimates
    avrg_words_per_paper = 10000
    avrg_words_per_book = 100000
    
    words = (papers*avrg_words_per_paper + books*avrg_words_per_book )
    
    # quick text of 27 million words from a few books
    sample_words = 27809550
    sample_bytes = 158824661
    sample_bytes_comp = 28839837 # using zpaq -m5
    
    bytes_per_word = sample_bytes/sample_words
    byte_comp_ratio = sample_bytes_comp/sample_bytes
    word_comp_ratio = bytes_per_word*byte_comp_ratio
    
    print("total:", words*bytes_per_word*1e-12, "TB") # total: 30.10238345855199 TB
    print("compressed:", words*word_comp_ratio*1e-12, "TB") # compressed: 5.466077036085319 TB
So uncompressed ~30 TB and compressed ~5.5 TB of data.

That fits on three 2TB micro SD cards, which you could buy for a total of 750$ from SanDisk.

Re: How large are large language models?

#140

Post by Tostino »

crazygringo wrote: post #134
pessimizer wrote: post #123

Earlier quoted context omitted.

Really? I'd assume that an LLM would deduplicate Wikipedia into something much smaller than 25GB. That's its only job.

> That's its only job. The vast, vast majority of LLM knowledge is not found in Wikipedia. It is definitely not its only job.

When trained on next word prediction with the standard loss function, by definition it is it's only job.
Post reply on HN