Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How large are large language models?

gist.github.com

Post reply on HN
Print view View original

101–110 of 160 posts

Re: How large are large language models?

#101

Post by tshaddox »

agumonkey wrote: post #55
ljoshua wrote: post #30

Less a technical comment and more just a mind-blown comment, but I still can’t get over just how much data is compressed into and available in these downloadable models. Yesterday I was on a plane with no WiFi, but had gemma3:12b downloaded through Ollama. Was playing around with it and showing my kids, and we fired history questions at it, questions about recent video games, and some animal fact questions. It wasn’t…

Intelligence is compression some say

Some say that. But what I value even more than compression is the ability to create new ideas which do not in any way exist in the set of all previously-conceived ideas.

Re: How large are large language models?

#102

Post by tshaddox »

Workaccount2 wrote: post #59
Wowfunhappy wrote: post #53

Earlier quoted context omitted.

A .zip is lossless compression. But we also have plenty of lossy compression algorithms. We've just never been able to use lossy compression on text.

>We've just never been able to use lossy compression on text. ...and we still can't. If your lawyer sent you your case files in the form of an LLM trained on those files, would you be comfortable with that? Where is the situation you would compress text with an LLM over a standard compression algo? (Other than to make an LLM). Other lossy compression targets known superfluous information. MP3 removes sounds we can't…

> If your lawyer sent you your case files in the form of an LLM trained on those files, would you be comfortable with that?

If the LLM-based compression method was well-understood and demonstrated to be reliable, I wouldn't oppose it on principle. If my lawyer didn't know what they were doing and threw together some ChatGPT document transfer system, of course I wouldn't trust it, but I also wouldn't trust my lawyer if they developed their own DCT-based lossy image compression algorithm.

Re: How large are large language models?

#103

Post by divbzero »

ljoshua wrote: post #30

Less a technical comment and more just a mind-blown comment, but I still can’t get over just how much data is compressed into and available in these downloadable models. Yesterday I was on a plane with no WiFi, but had gemma3:12b downloaded through Ollama. Was playing around with it and showing my kids, and we fired history questions at it, questions about recent video games, and some animal fact questions. It wasn’t…

The Encyclopædia Britannica has about 40,000,000 words [1] or about 0.25 GB if you assume 6 bytes per word. It’s impressive but not outlandish that an 8.1 GB file could encode a large swath of human information.

[1]: https://en.wikipedia.org/wiki/Encyclopædia_Britannica

Re: How large are large language models?

#104

Post by Wowfunhappy »

exe34 wrote: post #96
Wowfunhappy wrote: post #79

Earlier quoted context omitted.

> These models [...] have a huge “working memory” relative to us. [This is] indicative that vastly smarter models are going to be achieved fairly easily, with new insight. I don't think your second sentence logically follows from the first. Relative to us, these models: - Have a much larger working memory. - Have much more limited logical reasoning skills. To some extent, these models are able to use their superior w…

> But humans are able to reason with orders of magnitude less training data. Common belief, but false. You start learning from inside the womb. The data flow increases exponentially when you open your eyes and then again when you start manipulating things with your hands and mouth. > When you ask a model to "think about the problem step by step" to improve its reasoning, you are basically just giving it more opportun…

> Common belief, but false. You start learning from inside the womb. The data flow increases exponentially when you open your eyes and then again when you start manipulating things with your hands and mouth.

But you don't get data equal to the entire internet as a child!

> We do the same with children. At least I did it to my classmates when they asked me for help. I'd give them a hint, and ask them to work it out step by step from there. It helped.

And I do it with my students. I still think there's a difference in kind between when I listen to my students (or other adults) reason through a problem, and when I look at the output of an AI's reasoning, but I admittedly couldn't tell you what that is, so point taken. I still think the AI is relying far more heavily on its knowledge base.

Re: How large are large language models?

#105

Post by [deleted] »

loudmax wrote: post #19
dale_glass wrote: post #3

How big are those in terms of size on disk and VRAM size? Something like 1.61B just doesn't mean much to me since I don't know much about the guts of LLMs. But I'm curious about how that translates to computer hardware -- what specs would I need to run these? What could I run now, what would require spending some money, and what I might hope to be able to run in a decade?

Most of these models have been trained using 16-bit weights. So a 1 billion parameter model takes up 2 gigabytes. In practice, models can be quantized to smaller weights for inference. Usually, the performance loss going from 16 bit weights to 8 bit weights is very minor, so a 1 billion parameter model can take 1 gigabyte. Thinking about these models in terms of 8-bit quantized weights has the added benefit of making…

[deleted]

Re: How large are large language models?

#106

Post by lukeschlather »

This is a really nice writeup.

That said, there's an unstated assumption here that these truly large language models are the most interesting thing. The big players have been somewhat quiet but my impression from the outside is that OpenAI let a little bit leak with their behavior. They built an even larger model and it turned out to be disappointing so they quietly discontinued it. The most powerful frontier reasoning models may actually be smaller than the largest publicly available models.

Re: How large are large language models?

#107

Post by [deleted] »

divbzero wrote: post #103
ljoshua wrote: post #30

Less a technical comment and more just a mind-blown comment, but I still can’t get over just how much data is compressed into and available in these downloadable models. Yesterday I was on a plane with no WiFi, but had gemma3:12b downloaded through Ollama. Was playing around with it and showing my kids, and we fired history questions at it, questions about recent video games, and some animal fact questions. It wasn’t…

The Encyclopædia Britannica has about 40,000,000 words [1] or about 0.25 GB if you assume 6 bytes per word. It’s impressive but not outlandish that an 8.1 GB file could encode a large swath of human information. [1]: https://en.wikipedia.org/wiki/Encyclopædia_Britannica

[deleted]

Re: How large are large language models?

#108

Post by goatlover »

ToValueFunfetti wrote: post #90
goatlover wrote: post #81

Earlier quoted context omitted.

How well does that apply to robotics or animal intelligence? Manipulating the real world is more fundamental to human intelligence than compressing text.

Under the predictive coding model (and I'm sure some others), animal intelligence is also compression. The idea is that the early layers of the brain minimize how surprising incoming sensory signals are, so the later layers only have to work with truly entropic signal. But it has non-compression-based intelligence within those more abstract layers.

I just wonder if neuroscientists use that kind of model.

Re: How large are large language models?

#109

Post by jacobr1 »

Wowfunhappy wrote: post #104
exe34 wrote: post #96

Earlier quoted context omitted.

> But humans are able to reason with orders of magnitude less training data. Common belief, but false. You start learning from inside the womb. The data flow increases exponentially when you open your eyes and then again when you start manipulating things with your hands and mouth. > When you ask a model to "think about the problem step by step" to improve its reasoning, you are basically just giving it more opportun…

> Common belief, but false. You start learning from inside the womb. The data flow increases exponentially when you open your eyes and then again when you start manipulating things with your hands and mouth. But you don't get data equal to the entire internet as a child! > We do the same with children. At least I did it to my classmates when they asked me for help. I'd give them a hint, and ask them to work it out st…

There seems to be lots of mixed data points on this, but to some extent there is knowledge encoded into the evolutionary base state of the new human brain. Probably not directly as knowledge, but "primed" to quickly to establish relevant world models and certain types of reasoning with a small number of examples.

Re: How large are large language models?

#110

Post by AstroBen »

llSourcell wrote: post #98
netcan wrote: post #32

Earlier quoted context omitted.

Like both will be done. Idk what the roi is on adding video data to the text models, but it's presumably lower than text. There are just a lot of avenues to try at this point.

no its not lower than text, its higher ROI than text for understanding the physics of the world, which is exactly what videos are better at than text when it comes to training data

Does that transfer, though? I'm not sure we can expect its ability to approximate physics in video form would transfer to any other mode (text, code, problem solving etc)
Post reply on HN