Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How large are large language models?

gist.github.com

Post reply on HN
Print view View original

81–90 of 160 posts

Re: How large are large language models?

#81

Post by goatlover »

agumonkey wrote: post #55
ljoshua wrote: post #30

Less a technical comment and more just a mind-blown comment, but I still can’t get over just how much data is compressed into and available in these downloadable models. Yesterday I was on a plane with no WiFi, but had gemma3:12b downloaded through Ollama. Was playing around with it and showing my kids, and we fired history questions at it, questions about recent video games, and some animal fact questions. It wasn’t…

Intelligence is compression some say

How well does that apply to robotics or animal intelligence? Manipulating the real world is more fundamental to human intelligence than compressing text.

Re: How large are large language models?

#82

Post by [deleted] »

rain1 wrote: post #72
rybosome wrote: post #45

Earlier quoted context omitted.

Agreed, especially when in this context of training a smaller model on a larger model’s outputs. Distillation is generally accepted as an effective technique. This is exactly what I did in a previous role, fine-tuning Llama and Mistral models on a mix of human and GPT-4 data for a domain-specific task. Adding (good) synthetic data definitely increased the output quality for our tasks.

Yes but just purely in terms of entropy, you can't make a model better than GPT-4 by training it on GPT-4 outputs. The limit you would converge towards is GPT-4.

[deleted]

Re: How large are large language models?

#83

Post by tantalor »

rain1 wrote: post #74
kamranjon wrote: post #61

This is somehow missing the Gemma and Gemini series of models from Google. I also think that not mentioning the T5 series of models is strange from a historical perspective because they sort of pioneered many of the concepts in transfer learning and kinda kicked off quite a bit of interest in this space.

The Gemma models are too small to be included in this list. You're right the T5 stuff is very important historically but they're below 11B and I don't have much to say about them. Definitely a very interesting and important set of models though.

> too small

Eh?

* Gemma 1 (2024): 2B, 7B

* Gemma 2 (2024): 2B, 9B, 27B

* Gemma 3 (2025): 1B, 4B, 12B, 27B

This is the same range as some Llama models which you do mention.

> important historically

Aren't you trying to give a historical perspective? What's the point of this?

Re: How large are large language models?

#84

Post by simonw »

rain1 wrote: post #72
rybosome wrote: post #45

Earlier quoted context omitted.

Agreed, especially when in this context of training a smaller model on a larger model’s outputs. Distillation is generally accepted as an effective technique. This is exactly what I did in a previous role, fine-tuning Llama and Mistral models on a mix of human and GPT-4 data for a domain-specific task. Adding (good) synthetic data definitely increased the output quality for our tasks.

Yes but just purely in terms of entropy, you can't make a model better than GPT-4 by training it on GPT-4 outputs. The limit you would converge towards is GPT-4.

A better way to think about synthetic data is to consider code. With code you can have an LLM generate code with tests, then confirm that the code compiles and the tests pass. Now you have semi-verified new code you can add to your training data, and training on that will help you get better results for code even though it was generated by a "less good" LLM.

Re: How large are large language models?

#85

Post by stronglikedan »

ljoshua wrote: post #30

Less a technical comment and more just a mind-blown comment, but I still can’t get over just how much data is compressed into and available in these downloadable models. Yesterday I was on a plane with no WiFi, but had gemma3:12b downloaded through Ollama. Was playing around with it and showing my kids, and we fired history questions at it, questions about recent video games, and some animal fact questions. It wasn’t…

I've been doing the AI course on Brilliant lately, and it's mindblowing the techniques that they come up with to compress the data.

Re: How large are large language models?

#86

Post by Workaccount2 »

esafak wrote: post #73
Workaccount2 wrote: post #65

Earlier quoted context omitted.

Yes, and we all know (ask teachers) how reliable those summaries are. They are randomly lossy, which makes them unsuitable for any serious work. I'm not arguing that LLMs don't compress data, I am arguing that they are technically compression tools, but not colloquially compression tools, and the overlap they have with colloquial compression tools is almost zero.

> They are randomly lossy, which makes them unsuitable for any serious work. Ask ten people and they'll give ten different summaries. Are humans unsuitable too?

Yes, which is why we write things down, and when those archives become too big we use lossless compression on them, because we cannot tolerate a compression tool that drops the street address of a customer or even worse, hallucinates a slightly different one.

Re: How large are large language models?

#87

Post by alternatex »

tomkaos wrote: post #68
ljoshua wrote: post #30

Less a technical comment and more just a mind-blown comment, but I still can’t get over just how much data is compressed into and available in these downloadable models. Yesterday I was on a plane with no WiFi, but had gemma3:12b downloaded through Ollama. Was playing around with it and showing my kids, and we fired history questions at it, questions about recent video games, and some animal fact questions. It wasn’t…

Same thing with image model. 4 Go stable diffusion model can draw and represent anything humanity know.

How about a full glass of wine? Filled to the brim.

Re: How large are large language models?

#88

Post by antonvs »

Wowfunhappy wrote: post #79
Nevermark wrote: post #70

Earlier quoted context omitted.

It blows my mind that I can ask for 50 synonyms, instantly get a great list with great meaning summaries. Then ask for the same list sorted and get that nearly instantly, These models have a short time context for now, but they already have a huge “working memory” relative to us . It is very cool. And indicative that vastly smarter models are going to be achieved fairly easily, with new insight. Our biology has had t…

> These models [...] have a huge “working memory” relative to us. [This is] indicative that vastly smarter models are going to be achieved fairly easily, with new insight. I don't think your second sentence logically follows from the first. Relative to us, these models: - Have a much larger working memory. - Have much more limited logical reasoning skills. To some extent, these models are able to use their superior w…

> Have much more limited logical reasoning skills.

Relative to the best humans, perhaps, but I seriously doubt this is true in general. Most people I work with couldn’t reason nearly as well through the questions I use LLMs to answer.

It’s also worth keeping in mind that having a different approach to reasoning is not necessarily equivalent to a worse approach. Watch out for cherry-picking the cons of its approach and ignoring the pros.

Re: How large are large language models?

#89

Post by antonvs »

Workaccount2 wrote: post #59
Wowfunhappy wrote: post #53

Earlier quoted context omitted.

A .zip is lossless compression. But we also have plenty of lossy compression algorithms. We've just never been able to use lossy compression on text.

>We've just never been able to use lossy compression on text. ...and we still can't. If your lawyer sent you your case files in the form of an LLM trained on those files, would you be comfortable with that? Where is the situation you would compress text with an LLM over a standard compression algo? (Other than to make an LLM). Other lossy compression targets known superfluous information. MP3 removes sounds we can't…

> LLM's kind of do their own thing, and the data you get back out of them is correct, incorrect, or dangerously incorrect (i.e. is plausible enough to be taken as correct), with no algorithmic way to discern which is which.

Exactly like information from humans, then?

Re: How large are large language models?

#90

Post by ToValueFunfetti »

goatlover wrote: post #81
agumonkey wrote: post #55

Earlier quoted context omitted.

Intelligence is compression some say

How well does that apply to robotics or animal intelligence? Manipulating the real world is more fundamental to human intelligence than compressing text.

Under the predictive coding model (and I'm sure some others), animal intelligence is also compression. The idea is that the early layers of the brain minimize how surprising incoming sensory signals are, so the later layers only have to work with truly entropic signal. But it has non-compression-based intelligence within those more abstract layers.
Post reply on HN