Less a technical comment and more just a mind-blown comment, but I still can’t get over just how much data is compressed into and available in these downloadable models. Yesterday I was on a plane with no WiFi, but had gemma3:12b downloaded through Ollama. Was playing around with it and showing my kids, and we fired history questions at it, questions about recent video games, and some animal fact questions. It wasn’t…
How large are large language models?
121–130 of 160 posts
Re: How large are large language models?
#122Less a technical comment and more just a mind-blown comment, but I still can’t get over just how much data is compressed into and available in these downloadable models. Yesterday I was on a plane with no WiFi, but had gemma3:12b downloaded through Ollama. Was playing around with it and showing my kids, and we fired history questions at it, questions about recent video games, and some animal fact questions. It wasn’t…
Re: How large are large language models?
#123Earlier quoted context omitted.
For reference (according to Google): > The English Wikipedia, as of June 26, 2025, contains over 7 million articles and 63 million pages. The text content alone is approximately 156 GB, according to Wikipedia's statistics page. When including all revisions, the total size of the database is roughly 26 terabytes (26,455 GB)
Wikipedia itself describes its size as ~25GB without media [0]. And it's probably more accurate and with broader coverage in multiple languages compared to the LLM downloaded by the GP. https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia
Re: How large are large language models?
#124Less a technical comment and more just a mind-blown comment, but I still can’t get over just how much data is compressed into and available in these downloadable models. Yesterday I was on a plane with no WiFi, but had gemma3:12b downloaded through Ollama. Was playing around with it and showing my kids, and we fired history questions at it, questions about recent video games, and some animal fact questions. It wasn’t…
Intelligence is compression some say
Re: How large are large language models?
#125Earlier quoted context omitted.
Intelligence is compression some say
Some say that. But what I value even more than compression is the ability to create new ideas which do not in any way exist in the set of all previously-conceived ideas.
Its good enough that it has changed my mind about the fundamental utility of LLMs for coding in non-Javascript complexity regimes.
But its still not an expert programmer, not by a million miles, there is no way I could delegate my job to it (and keep my job). So there's some interesting boundary that's different than I used to think.
I think its in the vicinity of "how much precedent exists for this thought or idea or approach". The things I bring to the table in that setting have precedent too, but much more tenuously connected to like one clear precedent on e.g. GitHub, because if the thing I need was on GitHub I would download it.
Re: How large are large language models?
#126Earlier quoted context omitted.
It blows my mind that I can ask for 50 synonyms, instantly get a great list with great meaning summaries. Then ask for the same list sorted and get that nearly instantly, These models have a short time context for now, but they already have a huge “working memory” relative to us . It is very cool. And indicative that vastly smarter models are going to be achieved fairly easily, with new insight. Our biology has had t…
> These models [...] have a huge “working memory” relative to us. [This is] indicative that vastly smarter models are going to be achieved fairly easily, with new insight. I don't think your second sentence logically follows from the first. Relative to us, these models: - Have a much larger working memory. - Have much more limited logical reasoning skills. To some extent, these models are able to use their superior w…
But in evolutionary time frames, clearly those limits are lifting extraordinarily quickly. By many orders of magnitude.
And the point I made, that our limits were imposed by harsh biological energy and reward limits, vs. todays models (and their successors) which have access to relatively unlimited energy, and via sharing value with unlimited customers, unlimited rewards, stands.
It is a much simpler problem to improve digital cognition in a global ecosystem of energy production, instant communication and global application, than it was for evolution to improve an individual animals cognition in the limited resources of local habitats and their inefficient communication of advances.
Re: How large are large language models?
#127Re: How large are large language models?
#128Earlier quoted context omitted.
> These models [...] have a huge “working memory” relative to us. [This is] indicative that vastly smarter models are going to be achieved fairly easily, with new insight. I don't think your second sentence logically follows from the first. Relative to us, these models: - Have a much larger working memory. - Have much more limited logical reasoning skills. To some extent, these models are able to use their superior w…
> But humans are able to reason with orders of magnitude less training data. Common belief, but false. You start learning from inside the womb. The data flow increases exponentially when you open your eyes and then again when you start manipulating things with your hands and mouth. > When you ask a model to "think about the problem step by step" to improve its reasoning, you are basically just giving it more opportun…
Re: How large are large language models?
#129Earlier quoted context omitted.
> But humans are able to reason with orders of magnitude less training data. Common belief, but false. You start learning from inside the womb. The data flow increases exponentially when you open your eyes and then again when you start manipulating things with your hands and mouth. > When you ask a model to "think about the problem step by step" to improve its reasoning, you are basically just giving it more opportun…
> Common belief, but false. You start learning from inside the womb. The data flow increases exponentially when you open your eyes and then again when you start manipulating things with your hands and mouth. But you don't get data equal to the entire internet as a child! > We do the same with children. At least I did it to my classmates when they asked me for help. I'd give them a hint, and ask them to work it out st…
Given vision and the other senses, I’d argue that your average toddler has probably trained on more sensory information than the largest LLMs ever built long before they learn to talk.
Re: How large are large language models?
#130Less a technical comment and more just a mind-blown comment, but I still can’t get over just how much data is compressed into and available in these downloadable models. Yesterday I was on a plane with no WiFi, but had gemma3:12b downloaded through Ollama. Was playing around with it and showing my kids, and we fired history questions at it, questions about recent video games, and some animal fact questions. It wasn’t…