Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Teaching ChatGPT to speak my son’s invented language

szopa.medium.com

Post reply on HN
Print view View original

201–210 of 212 posts

Re: Teaching ChatGPT to speak my son’s invented language

#201

Post by int_19h »

samus wrote: post #49
JCharante wrote: post #15

Earlier quoted context omitted.

In my experience GPT4 poorly performs ROT13 but can do base64 decoding really well. A lot of the early jailbreaks used base64 to sneak tokens into prompts. How could it base64 decode but not reverse a string? That's very odd.

My guess: Decoding Base64 is easy because it's a 1:1 mapping between strings. Since it's not supposed to be an encryption or obfuscation, there must be huge lookup tables somewhere on the internet that it uses as Rosetta stones.

The other thing is, it's trained on a lot of HTML, which includes data: URLs that decode to JS and SVG (on which it is also trained). So that one transformation is probably the one that is really baked well into the weights now.

BTW it doesn't just decode it, it also encodes it quite happily - in real time, as it is producing the output, sometimes unprompted. I once had GPT-4, when asked to produce SVG, produce it in form of an - and when I copy-pasted and rendered it, it was a valid SVG file with shapes inside.

Re: Teaching ChatGPT to speak my son’s invented language

#202

Post by z3c0 »

joshribakoff wrote: post #94
z3c0 wrote: post #89

Earlier quoted context omitted.

Being able to discern its word soup as a description of what constitutes a game is more a demonstration of you understanding what a game is, not it. And therein lies the magic of ChatGPT: it knows nothing, but the projected significance of the token arrangements it creates tugs on what we know, and that causes a profound feeling akin to interacting with something intelligent.

Lets rephrase this differently, no one can prove whether it “understands” or not, and the same for humans. No one can prove whether consciousness is an “illusion” for humans. Your brain might just use embeddings, too. If you ask an average person if children understand what a game is, “yes” would be a common answer, but just like an AI model it may not have enough focus to play the game properly. I believe this is th…

You seem to be conflating solipsism with epistemology. At the end of the day, regardless of whether you're dealing with a child, a bot, or a peer, you would never use the question "describe how to play chess" as a measure of understanding if the chosen entity understands how to play chess; you would just play chess with them. Such an inquiry would only probe if one understands how to explain how to play chess, which is not the same. One should expect an LLM model of being able to regurgitate a description of playing chess. One could even expect an LLM to regurgitate common strategies in chess, enough to fool some into thinking it understands what is happening amidst a posed match. Where it all falls apart is where the rubber meets the pavement. Get an LLM to teach your child chess, with no intervention. You'll learn rather quickly which one is capable of understanding, and which one isn't.

Re: Teaching ChatGPT to speak my son’s invented language

#203

Post by sebzim4500 »

guenthert wrote: post #182
sebzim4500 wrote: post #161

Earlier quoted context omitted.

> repeated claims that it is just memorization and collaging I don't think anyone who has actually used GPT-4 for more than a few minutes ever believed this. It was just a trendy thing to say for a while.

It's rather said by people who have a rough understanding of how ML is implemented, rather then those who are mesmerized by its output.

Yeah, I think it's one of those cases where a little knowledge is a dangerous thing.

Re: Teaching ChatGPT to speak my son’s invented language

#204

Post by jychang »

leobg wrote: post #185
Jiocus wrote: post #172

Earlier quoted context omitted.

The psychology of split-brain is a nice gateway to 'The Bicameral Mind', a major theme about how the conception of the AI's works in the series Westworld. Nice!

Thank you. I am, in fact, reading it. Fascinating read. (How come all the best books were written in the 70s? Just survivership bias?)

Because the 1970s had bad science that was poorly replicated, especially in psychology.

Just because a theory sounds nice and seems to make sense doesn’t mean it’s scientific

Re: Teaching ChatGPT to speak my son’s invented language

#205

Post by jychang »

tel wrote: post #56
kloch wrote: post #33

Earlier quoted context omitted.

> you can supercharge accuracy to basically 100% for up to 13 digit addition and >90% after. Is the ~13 digit limitation due to the model itself (how/how well it was trained) or simply the use of double precision in the model weights (which maxes out around 15 digits of precision).

In order for it to be the second you'd need to assume that at least some part of the critical reasoning of the LLM involves storing the data of the number in a single model activation. This is pretty unlikely, as models tend to store information across many activations simultaneously. I don't know this for a fact, you'd need to do brain surgery on GPT-4 and it'd be hard even in that case, but most studies of ANN proc…

There are exceptions

https://www.lesswrong.com/posts/cgqh99SHsCv3jJYDS/we-found-a...

Re: Teaching ChatGPT to speak my son’s invented language

#206

Post by jurgemaister »

Enginerrrd wrote: post #167
Buttons840 wrote: post #58

Earlier quoted context omitted.

Being able to learn within context, without updating weights is amazing. Imagine how much more efficient and/or powerful it could be if we found a way to update the weights in real time.

Yeah I've been thinking about that a lot within the context of self-driving cars. What if we could take the rare but significant examples of model confusion, and then... explain to the model what it did wrong just as we would to a human. If it could then incorporate that information into future weights, that would be game changer in a LOT of fields where it's been hard to bridge the gap.

Tesla does this with shadow testing[1]

[1]https://www.forbes.com/sites/bradtempleton/2019/04/29/teslas...

Re: Teaching ChatGPT to speak my son’s invented language

#207

Post by hnfong »

int_19h wrote: post #195
sebzim4500 wrote: post #161

Earlier quoted context omitted.

> repeated claims that it is just memorization and collaging I don't think anyone who has actually used GPT-4 for more than a few minutes ever believed this. It was just a trendy thing to say for a while.

You can find people making such claims on HN, and some of them also claim that it is based on their personal experience with GPT-4. It boggles my mind, as well.

Superiority of human intelligence must not be questioned.

Re: Teaching ChatGPT to speak my son’s invented language

#208

Post by hnfong »

JKCalhoun wrote: post #102
felipemnoa wrote: post #31

Earlier quoted context omitted.

I wonder if that is how our brain produces dreams? The guardrails are down so it will just start producing ridiculous and/or implausible things. Edit: It almost seems like you are anthropomorphizing it. It is just a program doing what it's supposed to be doing: to predict the next token based on its weights. Nothing more, nothing less. It does give the illusion of intelligence. Pretty soon, though, we may not be able…

> It is just a program doing what it's supposed to be doing: to predict the next token based on its weights. Nothing more, nothing less. Every time I see a comment along these lines it gives me pause: there is a built-in assumption that each of us is somehow doing something more than this. I'm not convinced. I've heard people refer to some of our instinctive behaviors as due to "our lizard brain", suggesting that our…

To add to your point, note that a language model that "predicts the next token" accurately (as in predicting what human text would say) would pass the Turing test by definition.

One may argue what passing the Turing test means, but at least by definition it is mimicking human intelligence in some way.

Re: Teaching ChatGPT to speak my son’s invented language

#209

Post by tel »

jychang wrote: post #205
tel wrote: post #56

Earlier quoted context omitted.

In order for it to be the second you'd need to assume that at least some part of the critical reasoning of the LLM involves storing the data of the number in a single model activation. This is pretty unlikely, as models tend to store information across many activations simultaneously. I don't know this for a fact, you'd need to do brain surgery on GPT-4 and it'd be hard even in that case, but most studies of ANN proc…

There are exceptions https://www.lesswrong.com/posts/cgqh99SHsCv3jJYDS/we-found-a...

Sort of. I hadn't read that one and just skimmed it, but I think it falls into the same category of "while we can find compelling activation patterns even at a single neuron level, it's hard to rule out other representations".

On the flip side is Chris Olah's research (e.g. https://distill.pub/2017/feature-visualization/) which maps a single neuron's activation into the output space (by optimizing for outputs which would have arising from high activations of given neurons). If we take this seriously and call these outputs the "favored" output of an individual neuron, we might believe we've characterized that neuron, but it's really still the entire downstream computation from that neuron and there are still substantial overlaps and redundancies.

These are exactly where I'm getting the sense that the standard rule should be that neural nets have very distributed representations.

Even more to the point, these analyses tend to focus on "binary" questions, either a feature is present or not. Or, said another way, they implicitly treat activation as a "vote", where greater activation implies a greater component weighting of that which that neuron represents.

This, to my mind, flies in the face of GPT-n doing arithmetic, though. I'd lean much closer to the argument that each individual quantity is represented as a neuron as opposed to each quantity is represented continuously by the actual activation _value_.

This "voting" representation is also reinforced by the common use of tanh-like activation functions. There's a long history of having neuron activations be probabilistic/logit-like quantities.

Of course, I cannot speak for certain about any neural net and especially not for GPT-3/4. I just have a hard time taking as the null hypothesis that it is storing magnitudes of numbers in a way that is directly affected by the float precision in use.

Re: Teaching ChatGPT to speak my son’s invented language

#210

Post by ravi-delia »

xwdv wrote: post #106
ravi-delia wrote: post #105

Earlier quoted context omitted.

Clearly GPT doesn't "know" things in the way we do- I'd argue that there's probably a little bit of world modeling in there, but piecemeal and crude at best. Only so much you can do with such limited iteration. However, we must acknowledge that there are evidently some tasks that don't require knowledge the way we usually think about it. For a human, being able to talk about a game stems from the same place as our ab…

I’ll give you an example to think about in human terms. In political discussions, a lot of humans don’t really know what the fuck they’re talking about, but they do know what to say in response to certain stimulus. They know the talking points, the key phrases, the terms, they parrot these back to you when you provoke them to say it. They are basically human sentence predictors. Stuff comes out of their mouth based o…

That is indeed an example, but it still isn't an explanation. Sure, humans can also generate output without understanding- I've done it myself for my undergrad, throwing words together at 2am to make a deadline. I think quite a few people have remarked that LLMs seem to write like a human that isn't paying attention, which squares with what you said.

But the question remains why this is possible! A good enough pure predictor could play novel games or devise new theorems, but GPT absolutely can't. It can however give confident explanations, as well as write passable code and occasionally even do some novel problem solving (simple, impressive only because it comes from a computer, but still there). The question of why it can do some things and not others is interesting, and can't be swept under the rug just by reiterating that it's a predictor.

Does the structure of language really do such a good job of conveying information that GPT can operate on it blindly and get results, Blindsight style? Is composing prose far easier than we expect, leaving the bulk of the model free to do a tiny amount of "reasoning" that we find unjustly impressive because of how well it's presented? Is it handicapped primarily by the fact that it can only carry out extremely short computations, and can't be trained to use chain-of-reasoning to get around the limitation? We have no idea what, if any, inherent limitations predictive models have. We have no idea why GPT-sized models are good at the things they are, and bad at the things they aren't.

Post reply on HN