Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How large are large language models?

gist.github.com

Post reply on HN
Print view View original

141–150 of 160 posts

Re: How large are large language models?

#141

Post by arcticbull »

MPSimmons wrote: post #44
rain1 wrote: post #36

Earlier quoted context omitted.

It's extremely interesting how powerful a language model is at compression. When you train it to be an assistant model, it's better at compressing assistant transcripts than it is general text. There is an eval which I have a lot of interested in and respect for https://huggingface.co/spaces/Jellyfish042/UncheatableEval called UncheatableEval, which tests how good of a language model an LLM is by applying it on a ran…

Agreed. It's basically lossy compression for everything it's ever read. And the quantization impacts the lossiness, but since a lot of text is super fluffy, we tend not to notice as much as we would when we, say, listen to music that has been compressed in a lossy way.

I've been referring to LLMs as JPEG for all the world's data, and people have really started to come around to it. Initially most folks tended to outright reject this comparison.

Re: How large are large language models?

#142

Post by [deleted] »

jacobr1 wrote: post #111
Wowfunhappy wrote: post #79

Earlier quoted context omitted.

> These models [...] have a huge “working memory” relative to us. [This is] indicative that vastly smarter models are going to be achieved fairly easily, with new insight. I don't think your second sentence logically follows from the first. Relative to us, these models: - Have a much larger working memory. - Have much more limited logical reasoning skills. To some extent, these models are able to use their superior w…

> And by the way, we are out of new training data to give the models. Only easily accessible text data. We haven't really started using video at scale yet for example. It looks like data for specific tasks goes really far too ... for example agentic coding interactions aren't something that has generally been captured on the internet. But capturing interactions with coding agents, in combination with the base-trainin…

[deleted]

Re: How large are large language models?

#143

Post by wahnfrieden »

inopinatus wrote: post #131
sharkjacobs wrote: post #80

Earlier quoted context omitted.

better point of reference might be pages-articles-multistream.xml.bz2 (current pages without edit/revision history, no talk pages, no user pages) which is 20GB https://en.wikipedia.org/wiki/Wikipedia:Database_download#Wh... ?

this is a much more deserving and reliable candidate for any labels regarding the breadth of human knowledge.

it barely touches the surface

Re: How large are large language models?

#144

Post by dcl »

mr_toad wrote: post #121
ljoshua wrote: post #30

Less a technical comment and more just a mind-blown comment, but I still can’t get over just how much data is compressed into and available in these downloadable models. Yesterday I was on a plane with no WiFi, but had gemma3:12b downloaded through Ollama. Was playing around with it and showing my kids, and we fired history questions at it, questions about recent video games, and some animal fact questions. It wasn’t…

I will never tire of pointing out that machine learning models are compression algorithms, not compressed data.

ML algorithms are compression algorithms, the trained models are compressed data.

Re: How large are large language models?

#145

Post by simonw »

arcticbull wrote: post #141
MPSimmons wrote: post #44

Earlier quoted context omitted.

Agreed. It's basically lossy compression for everything it's ever read. And the quantization impacts the lossiness, but since a lot of text is super fluffy, we tend not to notice as much as we would when we, say, listen to music that has been compressed in a lossy way.

I've been referring to LLMs as JPEG for all the world's data, and people have really started to come around to it. Initially most folks tended to outright reject this comparison.

Ted Chiang wrote a great piece about that: https://www.newyorker.com/tech/annals-of-technology/chatgpt-...

I think it's a solid description for a raw model, but it's less applicable once you start combining an LLM with better context and tools.

What's interesting to me isn't the stuff the LLM "knows" - it's how well an LLM system can serve me when combined with RAG and tools like web search and access to a compiler.

The most interesting developments right now are models like Gemma 3n which are designed to have as much capability as possible without needing a huge amount of "facts" baked into them.

Re: How large are large language models?

#146

Post by ethan_smith »

dale_glass wrote: post #3

How big are those in terms of size on disk and VRAM size? Something like 1.61B just doesn't mean much to me since I don't know much about the guts of LLMs. But I'm curious about how that translates to computer hardware -- what specs would I need to run these? What could I run now, what would require spending some money, and what I might hope to be able to run in a decade?

As a rule of thumb, each billion parameters requires about 4GB of VRAM in FP16 (2 bytes per parameter), so a 7B model needs ~28GB, 70B needs ~280GB, while the 405B models need ~1.6TB of VRAM - though quantization can reduce this by 2-4x (4-bit models use only ~0.5GB per billion parameters).

Re: How large are large language models?

#147

Post by 1718627440 »

Wowfunhappy wrote: post #53
Workaccount2 wrote: post #49

Earlier quoted context omitted.

I don't like the term "compression" used with transformers because it gives the wrong idea about how they function. Like that they are a search tool glued onto a .zip file, your prompts are just fancy search queries, and hallucinations are just bugs in the recall algo. Although strictly speaking they have lots of information in a small package, they are F-tier compression algorithms because the loss is bad, unpredict…

A .zip is lossless compression. But we also have plenty of lossy compression algorithms. We've just never been able to use lossy compression on text.

SMS codes are kind of a lossy text-compression.

Re: How large are large language models?

#148

Post by 1718627440 »

mjburgess wrote: post #7
dale_glass wrote: post #3

How big are those in terms of size on disk and VRAM size? Something like 1.61B just doesn't mean much to me since I don't know much about the guts of LLMs. But I'm curious about how that translates to computer hardware -- what specs would I need to run these? What could I run now, what would require spending some money, and what I might hope to be able to run in a decade?

At 1byte/param that's 1.6GB (f8), at 2 bytes (f16) that's 2.3GB -- but there's other space costs beyond loading the parameters for the GPU. So a rule of thumb is ~4x parameter count. So round up, 2B -> 2*4 = 8GB VRAM

That sounds about the size of a modern browser (aka. any Electron et al. application)

Re: How large are large language models?

#149

Post by TeMPOraL »

ben_w wrote: post #50
TeMPOraL wrote: post #20

Earlier quoted context omitted.

> I read somewhere, but cannot find the source anymore, that all written text prior to this century was approx 50MB. (Might be misquoted as don't have source anymore). 50 MB feels too low, unless the quote meant text up until the 20th century , in which case it feels much more believable. In terms of text production and publishing, we're still riding an exponent, so a couple orders of magnitude increase between 1899…

Even by the start of the 20th century, 50 MB is definitely far too low. Any given English translation of Bible is by itself something like 3-5 megabytes of ASCII; the complete works of Shakespeare are about 5 megabytes; and I think (back of the envelope estimate) you'd get about the same again for what Arthur Conan Doyle wrote before 1900. I can just about believe there might have been only ten thousand Bible-or-Shak…

You and 'jerf make a fair point. Assuming you both are right, let's take jerf's estimate (which I now feel is right):

> 50MB feels like "all the 'ancient' text we have" maybe, as measured by the size of the original content and not counting copies

and yours - counting up court documents, newspapers, encyclopaedias, and I guess I'd add various letters to it (quite a lot survived to this day), and science[0], let's give it 1000x my estimate, so 50GB.

For the present, comments upthread give estimates that are in hundreds of terabytes to petabyte range. I'd say that, including deduplication, 50TB would be a conservative value. That's still 1000x of what you estimate for year 1900!

The exponent is going strong.

Thanks both of you for giving me a better picture of it.

--

[0] - I entirely forgot about https://en.wikipedia.org/wiki/Royal_Society!

Re: How large are large language models?

#150

Post by antonvs »

exe34 wrote: post #97
antonvs wrote: post #88

Earlier quoted context omitted.

> Have much more limited logical reasoning skills. Relative to the best humans, perhaps, but I seriously doubt this is true in general. Most people I work with couldn’t reason nearly as well through the questions I use LLMs to answer. It’s also worth keeping in mind that having a different approach to reasoning is not necessarily equivalent to a worse approach. Watch out for cherry-picking the cons of its approach an…

> Relative to the best humans, For some reason, the bar for AI is always against the best possible human, right now.

It seems that 90% of discussion about AI boils down to people who feel threatened by it in some way, and are lashing out in irrational ways as a result. (Source for 90% figure: Sturgeon's Law.)
Post reply on HN