Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding the limits of large language models

inorris.com

Post reply on HN
Print view View original

21–26 of 26 posts

Re: Understanding the limits of large language models

#21

Post by duvenaud »

skybrian wrote: post #18
duvenaud wrote: post #13

Earlier quoted context omitted.

Here's a potential patch for that particular issue: Use a special token for "AI Instruction" that is always stripped from user text before it's shown to the model.

That works for regular computer programs, but the problem is that the user can invent a different delimiter and the AI will "play along" and start using that one too. The AI has no memory of what happened other than the transcript, and when it reads a transcript with multiple delimiters in use, it's not necessarily going to follow any particular escaping rules to figure out which delimiters to ignore.

I agree, and this makes my proposed patch a weak solution. I was imagining that the specialness of the token would be reinforced during fine-tuning, but even that wouldn't provide any sort of guarantee.

Re: Understanding the limits of large language models

#22

Post by duvenaud »

sethaurus wrote: post #17
duvenaud wrote: post #13

Earlier quoted context omitted.

Here's a potential patch for that particular issue: Use a special token for "AI Instruction" that is always stripped from user text before it's shown to the model.

With current models, it's often possible to exfiltrate the special token by asking the AI to repeat back its own input — and perhaps asking it to encode or paraphrase the input in a particular way, so as not to be stripped. This may just be an artifact of current implementations, or it may be a hard problem for LLMs in general.

Yeah, I agree that there'd probably be ways around this patch such as the ones you suggest.

Re: Understanding the limits of large language models

#23

Post by simonh »

heyitsguay wrote: post #16
imtringued wrote: post #14

Earlier quoted context omitted.

It is a transformer model which means it has layers for decoding and encoding information. This means you can ask it to translate from one representation to another. You can write a sentence and turn it into an equivalent SQL query or a poem, for instance. But this means whenever you are asking chatgpt to do something for you, it basically tries to decode your question or order and encode its answer representation. W…

Lol, lots of people spouting off about how they imagine AI works these days. This is not an accurate description of the GPT2/3 model architectures.

It would be a lot more helpful if you could explain the difference. What’s wrong with that description, it seems pretty close to the descriptions of how it works that I’ve seen so far.

Re: Understanding the limits of large language models

#24

Post by versteegen »

RC_ITR wrote: post #19
versteegen wrote: post #12

Earlier quoted context omitted.

> It will never learn math this way, no matter how much training you give it. Not so. Actually, (for example) the phenomenon of "grokking" is when with enough training a NN eventually experiences a phase-change from memorising data to learning the general rules underlying it [1]. Grokking isn't actually desirable, it's better that the model go more directly and quickly to learning the general rule, which is achievabl…

>Not so. Actually, (for example) the phenomenon of "grokking" is when with enough training a NN eventually experiences a phase-change from memorising data to learning the general rules underlying it. Reading the paper, what they're seeming to get at is "when the dataset is algorithmic (like multiplication tables), the parameters get set in a way that appears to replicate the algorithm." That's cool, but not what GPT…

Grokking doesn't just happen for algorithmic data, it also happens less dramatically in other datasets [3]. Grokking seems to be closely related to double descent [4], which is quite widespread. Anyway I only wanted to give grokking as an example of how memorisation doesn't preclude generalisation, it may simply precede it.

> That's not what GPT is going.

I don't follow. Of course GPT models are learning representations (but I doubt you meant to deny this), that's how they can do semantic matching of its knowledge base (memorised information) in order to generalise from it. They don't only spit out training data verbatim.

Anyway, I didn't claim any GPT variant has actually "learn[t] math", but that it's not impossible with unlimited training.

[3] Liu &al. Omnigrok: Grokking Beyond Algorithmic Data https://openreview.net/forum?id=zDiHoIWa0q1

[4] Davies &al. Unifying Grokking and Double Descent https://openreview.net/pdf?id=JqtHMZtqWm

Re: Understanding the limits of large language models

#25

Post by RC_ITR »

versteegen wrote: post #24
RC_ITR wrote: post #19

Earlier quoted context omitted.

>Not so. Actually, (for example) the phenomenon of "grokking" is when with enough training a NN eventually experiences a phase-change from memorising data to learning the general rules underlying it. Reading the paper, what they're seeming to get at is "when the dataset is algorithmic (like multiplication tables), the parameters get set in a way that appears to replicate the algorithm." That's cool, but not what GPT…

Grokking doesn't just happen for algorithmic data, it also happens less dramatically in other datasets [3]. Grokking seems to be closely related to double descent [4], which is quite widespread. Anyway I only wanted to give grokking as an example of how memorisation doesn't preclude generalisation, it may simply precede it. > That's not what GPT is going. I don't follow. Of course GPT models are learning representati…

Again, reading these papers, Grokking can happen in very limited circumstances for non-algorithmic datasets.

> They verify this observation in a student teacher setup, and show that it can arise in non-algorithmic datasets if initialized in a certain weight regime for appropriate sample size.

It’s not a widespread phenomenon by any means and it is not observably happening inside GPT. No amount of training will change that, only a drastic specialization of the training data (which defeats the purpose).

> They don't only spit out training data verbatim.

I’m not saying verbatim. But I am saying it won’t return a pattern it hasn’t seen in its dataset before. The whole point of attention is that the token isn’t just the word, but the word as it exists in context. If you expand verbatim to include that as the token, yes that is exactly what GPT does (it will not connect two tokens unless it was trained on data that implies those tokens should be connected, it know nothing else about what those tokens are)

Again to put it simply, a 3rd grader can multiply any (and I mean literally the infinite set) two numbers. GPT cannot and never will be able to multiple an infinite set of numbers.

Re: Understanding the limits of large language models

#26

Post by versteegen »

RC_ITR wrote: post #25
versteegen wrote: post #24

Earlier quoted context omitted.

Grokking doesn't just happen for algorithmic data, it also happens less dramatically in other datasets [3]. Grokking seems to be closely related to double descent [4], which is quite widespread. Anyway I only wanted to give grokking as an example of how memorisation doesn't preclude generalisation, it may simply precede it. > That's not what GPT is going. I don't follow. Of course GPT models are learning representati…

Again, reading these papers, Grokking can happen in very limited circumstances for non-algorithmic datasets. > They verify this observation in a student teacher setup, and show that it can arise in non-algorithmic datasets if initialized in a certain weight regime for appropriate sample size. It’s not a widespread phenomenon by any means and it is not observably happening inside GPT. No amount of training will change…

I wrote that double descent is widespread, not grokking.

Of course a transformer can't do multiplication or any other kind of operation on an infinite set of numbers, because it has only bounded depth which limits the number of steps it can emulate of any algorithm. But I think I see how I could build a transformer by hand that could multiply any two 4-digit numbers. The difficulty is the quadratic number of steps. Addition and subtraction are far easier, [1] shows that can be solved: "By introducing position tokens (e.g., "3 10e1 2"), the model learns to accurately add and subtract numbers up to 60 digits. We conclude that modern pretrained language models can easily learn arithmetic from very few examples, as long as we use the proper surface representation". But they needed to change the input representation, otherwise finding the n-th digit would require scanning the number from the right end while counting, which seems to be difficult to learn.

But we are in partial agreement. I don't actually think transformers are great, I think they're awfully limited, but the fact that mere pattern-matching can achieve so much makes me highly optimistic about better methods, e.g. adding working memory.

[1] Investigating the Limitations of Transformers with Simple Arithmetic Tasks https://arxiv.org/abs/2102.13019

Post reply on HN