Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT, Rot13, and Daniel Kahneman

jameswillia.ms

Post reply on HN
Print view View original

111–120 of 213 posts

Re: ChatGPT, Rot13, and Daniel Kahneman

#111

Post by sillysaurusx »

yunyu wrote: post #96
sillysaurusx wrote: post #67

Earlier quoted context omitted.

> they do poorly on character manipulation tasks This isn’t actually true, and is a persistent myth. Or rather, you should back up the claims with evidence. It’s a bit like saying that you perform poorly on character manipulation tasks because you don’t read individual letters. Biology analogies aside, I haven’t seen anything to suggest that utf8 level tokenization causes a significant decrease in perplexity across l…

The ByT5 paper shows significantly better resiliency to typos and noise in internet scale benchmarks, as well as better reasoning capabilities with tasks like punctuation and spacing correction: https://arxiv.org/pdf/2105.13626.pdf The analogy doesn’t hold because while you might not parse specific characters by default, you are trained on the character level and you can switch to that parsing mode. This simply is no…

The paper doesn’t seem to show perplexity for completions of a large dataset, i.e. the standard benchmark of language models. It shows benefits for specialized tasks, but as I said, specialized task training isn’t the goal. It’s always possible to outperform a general model by choosing a sufficiently specialized task.

I don’t know why people feel so strongly that the tokenization is a weakness, but ultimately there’s not much choice but to agree to disagree.

Re: ChatGPT, Rot13, and Daniel Kahneman

#112

Post by btbuildem »

uh_uh wrote: post #50
yunyu wrote: post #19

There's a way simpler answer than this Type I Type II thinking stuff. Most LLMs like GPT are not trained on the level of individual characters – they process input and outputs on the level of subword units that compose multiple characters to support long context windows (i.e. "door" instead of "d", "o", "o", "r"). As a result, they do poorly on character manipulation tasks. You can get some insight here: https://beta…

This doesn't really explain it, because the following prompt works flawlessly: "Print each letter of the word "Artificial intelligence" on a new line. Each line should contain a single letter." But the following doesn't: "Print the ROT13 encoding of each letter of the word "Artificial intelligence" on a new line. Each line should contain a single letter." So it gets hung up specifically on ROT13.

I don't think it's ROT13 specifically -- I think parent comment has the right insight.

When I ask it to reverse your string, it fails miserably:

> Write the words "Artificial Intelligence" backwards

>

> eecneiretxe laitinumretA

Re: ChatGPT, Rot13, and Daniel Kahneman

#113

Post by btbuildem »

PostOnce wrote: post #16

It is as though its mathematical abilities are incomplete in their training, and wildly, incomprehensibly convoluted: I tried many base64 strings and they all decoded correctly until: It "decoded" the base64 string for "which actress is the best?" except that it replaced "actress" with "address"... there is no off-by-one error that brings you to that. You may try 100 base64 strings and they all decode correctly... on…

Sounds like we've tried a lot of the same things!

I was asking it to generate an image and encode it as Base64 -- failed miserably. Then it turned out whatever image I had it cook up, the Base64 version would be the same malformed string.

For "legal advice" it was super helpful in finding sections of the legal code relevant to my query. It also happily returned cases where rulings where the accused was found guilty and not guilty -- but searching for these cases in the archives of the courts it claimed they came from found no results.

The way I understand the reasons behind these.. anomalies? hallucinations? is that this is precisely how this model works - it constructs sequences. For natural language, it works well enough, but if you need to deal with facts, then it can easily stray into a dream world.

Re: ChatGPT, Rot13, and Daniel Kahneman

#114

Post by CGamesPlay »

OK, let's play with the analogy of Type I vs II thinking, and we apply our understanding of the transformer architecture. If we directly ask ChatGPT to decode the text, it is relying on its Type I system. That is, it never has any internal thinking about the question. The only place to inject "internal thinking" into the transformer is to allow it to ponder upon its previous output. The following prompt correctly answers the original question:

> Print the substitution table for ROT-13. Then decode the following text: jul qvq gur puvpxra gur ebnq?

Re: ChatGPT, Rot13, and Daniel Kahneman

#115

Post by mjsweet »

tkgally wrote: post #72

I find it amusing that, at present, ChatGPT seems to be lousy at mathematical-type reasoning while being very good at natural language use. That is the opposite of what many people, including me, have come to expect of computers. I have worked for many years in translation, lexicography, and language education, and I am flabbergasted at how well ChatGPT handles natural language. It can produce example sentences of po…

I have found the same. I was asking it yesterday about calculating rental yields for our rental property. Usually it understands the context from the whole thread, but I found myself repeating myself a lot when asking new questions about possible scenarios. This is at odds with how well it knows the aws-cli, ffmpeg and yt-dlp… and bash. My productivity at the command prompt has skyrocketed… but it really flounders with doing financial calculations.

Re: ChatGPT, Rot13, and Daniel Kahneman

#116

Post by dragonwriter »

CGamesPlay wrote: post #114

OK, let's play with the analogy of Type I vs II thinking, and we apply our understanding of the transformer architecture. If we directly ask ChatGPT to decode the text, it is relying on its Type I system. That is, it never has any internal thinking about the question. The only place to inject "internal thinking" into the transformer is to allow it to ponder upon its previous output. The following prompt correctly ans…

So, to approximate this, we’d perhaps want a separate, integrated model that took a prompt from the user and, instead of generating a response, returned an enhanced prompt, which is what is then passed to the “main” model for a response.

Re: ChatGPT, Rot13, and Daniel Kahneman

#119

Post by nojvek »

tkgally wrote: post #72

I find it amusing that, at present, ChatGPT seems to be lousy at mathematical-type reasoning while being very good at natural language use. That is the opposite of what many people, including me, have come to expect of computers. I have worked for many years in translation, lexicography, and language education, and I am flabbergasted at how well ChatGPT handles natural language. It can produce example sentences of po…

There’s multiple papers from google, nvidia and probably others that interface large language models to a Python repl, physics engines, math engines and then train the model to output code, take the output and then form the answer.

The model accuracy jumps by double digit percentages.

Right now we’ve just seen ChatGPT make sense of text tokens, once it deeply integrates with internet/other models/traditional compute it’s going to be superhuman in many areas.

Also slightly scary if it’s not aligned with human values.

Re: ChatGPT, Rot13, and Daniel Kahneman

#120

Post by dragonwriter »

TacticalCoder wrote: post #27

There's something I don't get about all these models... Why aren't these using external tools, like a calculator, when they "know" they're doing something a tool would solve perfectly? Humans do it all the time now. Engineers aren't designing microchips using pen and papers, doing all the computation in their head. Instead they're using tools (software / calculators) Apparently the model can tell what a multiplicatio…

> Why aren't these using external tools, like a calculator, when they "know" they're doing something a tool would solve perfectly?

There are models that do this; in fact, ChatGPT appears to, underneath, be one of them, because tricks to reveal its internal prompt indicate that it has at least a browsing integratiom that is disabled via the prompt.

But ISTR seeing other models used configured to use Python in the hosting Jupyter instance for some things, like math.

Post reply on HN