31–40 of 156 posts
This is really cool. My mind goes immediately to what sort of functions are being used to encode programming knowledge, and if they are also simple linear functions whether the standard library or other libraries can be directly uploaded into an LLMs brain as it evolves, without needing to go through a costly training or performance-destroying fine-tune. That's still a sci-fi ability today but it seems to be getting…
This is really cool. My mind goes immediately to what sort of functions are being used to encode programming knowledge, and if they are also simple linear functions whether the standard library or other libraries can be directly uploaded into an LLMs brain as it evolves, without needing to go through a costly training or performance-destroying fine-tune. That's still a sci-fi ability today but it seems to be getting…
That's a good point. It may be possible to directly upload predicate-type info into a LLM. This could be especially useful if you need to encode tabular data. Somewhere, someone probably read this and is thinking about how to export Excel or databases to an LLM. It's encouraging to see people looking inside the black box successfully. The other big result in this area was that paper which found a representation of a…
The external service vector database may require tight timings necessary to avoid stalling LLMs. To manage 20-50 tokens/sec, answers must arrive within 50-20ms.
And we cannot do this in real-time, pausing the transformer when a layer produces a query vector stalls the batch, so we need a way to predict queries (or embeddings) several tokens ahead of where they'd be useful and inject the context in when it's needed, and to know when to page it out.
I don't understand how a "CSV file/database/model" of 70,000,000,000 (70B) "parameters" of 4-bit weights (a 4 bit value can be 1 of 16 unique numbers) gets us an interactive LLM/GPT that is near-all-knowledgable on all topics/subjects. edit: did research, the 4-bit is just a "compression method", the model ends up seeing f32? > Quantization is the process of mapping 32-bit floating-point numbers (which are the weight…
If the model in question has a token embedding length of 1024, even if it was a 1 bit quantization, each token has 2^1024 possible values.
If the context length is 32,000 tokens, there are 32,000^2^1024 possible inputs.
Earlier quoted context omitted.
It doesn't, is the simple answer. The slightly more complicated one is that a compressed text dump of Wikipedia isn't even 70GB, and this is lossy compression of the internet.
Is there some sort of "LLM-on-Wikipedia" competition? ie: given "just wikipedia" what's the best score people can get on however these models are evaluated. I know that all the commercial ventures have a voracious data-input set, but it seems like there's room for dictionary.llm + wikipedia.llm + linux-kernel.llm and some sort of judging / bake-off for their different performance capabilities. Or does the training tr…
I have the same question.
Peter Norvig’s GOFAI Shakespeare generator example[1] (which is not an LLM) gets impressive results with little input data to go on. Does the leap to LLM preclude that kind of small input approach?
[1] link should be here because I assumed as I wrote the above that I would just turn it up with a quick google. Alas t’was not to be. Take my word for it, somewhere on t’internet is an excellent write up by Peter Norvig on LLM vs GOFAI (good old fashioned artificial intelligence)
Earlier quoted context omitted.
say the average LLM these days has a unique token (vocabulary) size of ~32,000 (not its context size, # of unique tokens it can pick between in a response. English words, punctuation, math, code, etc.) the 60-70B parameters of models is basically like... just stored patterns of "if these 10 tokens in a row input, then these 10 tokens in a row output score the highest" Is that a good summary? > The model uses its lear…
Your suggested scheme (assuming a mapping from 10 tokens to 10 tokens, with each token taking 2 bytes to store) would take (32000 * 20) * 2 bytes = 2.3e78 TiB of storage, or about 250 MiB per atom in the observable universe (1e82), prior to compression. I think it's more likely that LLMs are actually learning and understanding concepts as well as memorizing useful facts, than that LLMs have discovered a compression m…
I find this similar to what relation vectors do in word2vec: you can add a vector of "X of" and often get the correct answer. It could be that the principle is still the same, and transformers "just" build a better mapping of entities into the embedding space?
x = 0 means “fly”
x = 0.01 means “drive”
x = 0.02 means “purple”
x 1.5 means “hot”
Llms seem like a good compression mechanism. It blows my mind that I can have a copy of llama locally on my PC and have access to virtually the entire internet
Earlier quoted context omitted.
It doesn't, is the simple answer. The slightly more complicated one is that a compressed text dump of Wikipedia isn't even 70GB, and this is lossy compression of the internet.
say the average LLM these days has a unique token (vocabulary) size of ~32,000 (not its context size, # of unique tokens it can pick between in a response. English words, punctuation, math, code, etc.) the 60-70B parameters of models is basically like... just stored patterns of "if these 10 tokens in a row input, then these 10 tokens in a row output score the highest" Is that a good summary? > The model uses its lear…
Modern LLMs are able to transfer knowledge between different languages, so it's fair to assume that some mapping between human language and a more abstract internal representation happens at the input and output, instead of the model "operating" on English or Chinese or whatever language you talk with it. And once this exists, an internal "world model" (as in: a collection of facts and implications) isn't far, and seems to indeed be something most LLMs do. The reasoning on top of that world model is still very spotty though
I find this similar to what relation vectors do in word2vec: you can add a vector of "X of" and often get the correct answer. It could be that the principle is still the same, and transformers "just" build a better mapping of entities into the embedding space?
I think so. It’s hard for me to believe that the decision surfaces inside those models are really curved enough (like the folds of your brain) to really take advantage of FP32 numbers inside vectors: that is I just don’t believe it is x = 0 means “fly” x = 0.01 means “drive” x = 0.02 means “purple” but rather more like x 1.5 means “hot” which is one reason why quantization (often 1 bit) works. Also it is a reason why…