LLMs use a surprisingly simple mechanism to retrieve some stored knowledge
1–10 of 156 posts
Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge
#2> In one experiment, they started with the prompt “Bill Bradley was a” and used the decoding functions for “plays sports” and “attended university” to see if the model knows that Sen. Bradley was a basketball player who attended Princeton.
Why not just change the prompt?
Name, University attended, Sport played
Bill Bradley,Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge
#3> Linear functions, equations with only two variables and no exponents, capture the straightforward, straight-line relationship between two variables
Is this definition considering the output to be included in the set of variables? What a strange way to phrase it. Under this definition, I wonder what an equation with one variable is. Is a single constant an equation?
Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge
#4Llms seem like a good compression mechanism.
It blows my mind that I can have a copy of llama locally on my PC and have access to virtually the entire internet
Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge
#5I thought "fact" means truth.
Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge
#6I don't understand how a "CSV file/database/model" of 70,000,000,000 (70B) "parameters" of 4-bit weights (a 4 bit value can be 1 of 16 unique numbers) gets us an interactive LLM/GPT that is near-all-knowledgable on all topics/subjects.
edit: did research, the 4-bit is just a "compression method", the model ends up seeing f32?
> Quantization is the process of mapping 32-bit floating-point numbers (which are the weights in the neural network) to a much smaller bit representation, like 4-bit values, for storage and memory efficiency.
> Dequantization happens when the model is used (during inference or even training, if applicable). The 4-bit quantized weights are converted back into floating-point numbers that the model's computations are actually performed with. This is done using the scale and zero-point determined during the initial quantization, or through more sophisticated mapping functions that aim to preserve as much information as possible despite the reduced precision.
so what is the relationship to "parameters" and "# of unique tokens the model knows about (vocabulary size)"?
> At first glance, LLAMa only has a 32,000 vocabulary size and 65B parameters as compared to GPT-3,
> The 65 billion parameters in a model like LLAMA (or any large language model) essentially function as a highly intricate mapping system that determines how to respond to a given input based on the learned relationships between tokens in its training data.
Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge
#7I find this similar to what relation vectors do in word2vec: you can add a vector of "X of" and often get the correct answer.
It could be that the principle is still the same, and transformers "just" build a better mapping of entities into the embedding space?
Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge
#8I don't understand how a "CSV file/database/model" of 70,000,000,000 (70B) "parameters" of 4-bit weights (a 4 bit value can be 1 of 16 unique numbers) gets us an interactive LLM/GPT that is near-all-knowledgable on all topics/subjects. edit: did research, the 4-bit is just a "compression method", the model ends up seeing f32? > Quantization is the process of mapping 32-bit floating-point numbers (which are the weight…
It doesn't, is the simple answer.
The slightly more complicated one is that a compressed text dump of Wikipedia isn't even 70GB, and this is lossy compression of the internet.
Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge
#9Help me understand: when they say that the facts are stored as a linear function… are they saying that the LLM has a sort of N-dimensional “fact space” encoded into the model in some manner, where facts are embedded into the space as (points / hyperspheres / Voronoi manifolds / etc); and where recalling a fact is — at least in an abstract sense — the NN computing / remembering a key to use, and then doing a key-value lookup in this space?
If so: how do you embed a KV-store into an edge-propagated graphical model? Are there even any well-known techniques for doing that “by hand” right now?
(Also, fun tangent: isn't the "memory palace" memory technique, an example of human brains embedding facts into a linear function for easier retrieval?)
Re: LLMs use a surprisingly simple mechanism to retrieve some stored knowledge
#10I don't understand how a "CSV file/database/model" of 70,000,000,000 (70B) "parameters" of 4-bit weights (a 4 bit value can be 1 of 16 unique numbers) gets us an interactive LLM/GPT that is near-all-knowledgable on all topics/subjects. edit: did research, the 4-bit is just a "compression method", the model ends up seeing f32? > Quantization is the process of mapping 32-bit floating-point numbers (which are the weight…
It doesn't, is the simple answer. The slightly more complicated one is that a compressed text dump of Wikipedia isn't even 70GB, and this is lossy compression of the internet.
say the average LLM these days has a unique token (vocabulary) size of ~32,000 (not its context size, # of unique tokens it can pick between in a response. English words, punctuation, math, code, etc.)
the 60-70B parameters of models is basically like... just stored patterns of "if these 10 tokens in a row input, then these 10 tokens in a row output score the highest"
Is that a good summary?
> The model uses its learned statistical patterns to predict the probability of what comes next in a sequence of text.
based on what inputs?
1. previous tokens in the sequence from immediate context
2. tokens summarizing the overall topic/subject matter from the extended context
3. scoring of learned patterns from training
4. what else?