Big LLMs weights are a piece of history
antirez.com
Big LLMs weights are a piece of history
1–10 of 222 posts
Re: Big LLMs weights are a piece of history
#2That's really what these are: something analogous to JPEG for language, and queryable in natural language.
Tangent: I was thinking the other day: these are not AI in the sense that they are not primarily intelligence. I still don't see much evidence of that. What they do give me is superhuman memory. The main thing I use them for is search, research, and a "rubber duck" that talks back, and it's like having an intern who has memorized the library and the entire Internet. They occasionally hallucinate or make mistakes -- compression artifacts -- but it's there.
So it's more AM -- artificial memory.
Edit: as a reply pointed out: this is Vannevar Bush's Memex, kind of.
Re: Big LLMs weights are a piece of history
#3That's really what these are: something analogous to JPEG for language, and queryable in natural language. Tangent: I was thinking the other day: these are not AI in the sense that they are not primarily intelligence . I still don't see much evidence of that. What they do give me is superhuman memory. The main thing I use them for is search, research, and a "rubber duck" that talks back, and it's like having an inter…
I believe LLMs are both data and processing, but even humans reasoning is based in strong ways on existing knowledge. However, for the goal of the post, indeed it is the memorization that is the key value, and the fact that likely in the future sampling such models can be used to transfer the same knowledge to bigger LLMs, even if the source data is lost.
Re: Big LLMs weights are a piece of history
#4That's really what these are: something analogous to JPEG for language, and queryable in natural language. Tangent: I was thinking the other day: these are not AI in the sense that they are not primarily intelligence . I still don't see much evidence of that. What they do give me is superhuman memory. The main thing I use them for is search, research, and a "rubber duck" that talks back, and it's like having an inter…
I believe LLMs are both data and processing, but even humans reasoning is based in strong ways on existing knowledge. However, for the goal of the post, indeed it is the memorization that is the key value, and the fact that likely in the future sampling such models can be used to transfer the same knowledge to bigger LLMs, even if the source data is lost.
I'm not saying there is no latent reasoning capability. It's there. It just seems to be that the memory and lookup component is much more useful and powerful.
To me intelligence describes something much more capable than what I see in these things, even the bleeding edge ones. At least so far.
Re: Big LLMs weights are a piece of history
#5Re: Big LLMs weights are a piece of history
#6Earlier quoted context omitted.
I believe LLMs are both data and processing, but even humans reasoning is based in strong ways on existing knowledge. However, for the goal of the post, indeed it is the memorization that is the key value, and the fact that likely in the future sampling such models can be used to transfer the same knowledge to bigger LLMs, even if the source data is lost.
I'm not saying there is no latent reasoning capability. It's there. It just seems to be that the memory and lookup component is much more useful and powerful. To me intelligence describes something much more capable than what I see in these things, even the bleeding edge ones. At least so far .
I offer a POV that is in the middle: reasoning is powerful to evaluate which solution is better among N in the context. Memorization allows sampling of many competing ideas from the problem space, than the LLM picks the best, making chain of thoughts so effective. Of course zero shot reasoning also is a part of the story but somewhat weaker, exactly like we are not often able to spit the best solution before evaluation of the space (unless we are very accustomed to the specific problem).
Re: Big LLMs weights are a piece of history
#7That's really what these are: something analogous to JPEG for language, and queryable in natural language. Tangent: I was thinking the other day: these are not AI in the sense that they are not primarily intelligence . I still don't see much evidence of that. What they do give me is superhuman memory. The main thing I use them for is search, research, and a "rubber duck" that talks back, and it's like having an inter…
This is an excellent viewpoint.
Re: Big LLMs weights are a piece of history
#8That's really what these are: something analogous to JPEG for language, and queryable in natural language. Tangent: I was thinking the other day: these are not AI in the sense that they are not primarily intelligence . I still don't see much evidence of that. What they do give me is superhuman memory. The main thing I use them for is search, research, and a "rubber duck" that talks back, and it's like having an inter…
Having memory is fine but choosing the relevant parts requires intelligence
Re: Big LLMs weights are a piece of history
#9That's really what these are: something analogous to JPEG for language, and queryable in natural language. Tangent: I was thinking the other day: these are not AI in the sense that they are not primarily intelligence . I still don't see much evidence of that. What they do give me is superhuman memory. The main thing I use them for is search, research, and a "rubber duck" that talks back, and it's like having an inter…
Or 80 years to MVP memex
“Vannevar Bush's 1945 article "As We May Think". Bush envisioned the memex as a device in which individuals would compress and store all of their books, records, and communications, "mechanized so that it may be consulted with exceeding speed and flexibility".
Re: Big LLMs weights are a piece of history
#10That's really what these are: something analogous to JPEG for language, and queryable in natural language. Tangent: I was thinking the other day: these are not AI in the sense that they are not primarily intelligence . I still don't see much evidence of that. What they do give me is superhuman memory. The main thing I use them for is search, research, and a "rubber duck" that talks back, and it's like having an inter…
I've been looking at it as an "instant reddit comment". I can download a 10G or 80G compressed archive that basically contains the useful parts of the internet, and then I all can use it to synthesize something that is about as good and reliable as a really good reddit comment. Which is nifty. But honestly it's an incredible idea to sell that to businesses.