Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT, Rot13, and Daniel Kahneman

jameswillia.ms

Post reply on HN
Print view View original

11–20 of 213 posts

Re: ChatGPT, Rot13, and Daniel Kahneman

#11

Post by trh0awayman »

vessenes wrote: post #6
Temporary_31337 wrote: post #3

Earlier quoted context omitted.

Thing is chatGPT seems overconfident in its answers so unless you know the answer ahead of time you have no certainty that it is a correct math - try simple division question for example.

Some of this has to do with the likely prompts surrounding chatgpt - it's probably been instructed to be helpful, positive, etc. If you need it to be more honest / say no more, you just have to ask and reinforce. That said, ROT13 is a tough job for a tokenized LLM, because it doesn't think in terms of letters. chatGPT is down right now, so I can't test these, but I would guess that for ROT13, the following would work…

So now we know how to hide from the AI

Re: ChatGPT, Rot13, and Daniel Kahneman

#13

Post by FL410 »

volleygman180 wrote: post #5

I love how it eventually gives up and throws an "Internal Server error". Over the weekend, I was finding ChatGPT giving me incorrect answers for some things too. In one case, it would try to correct itself when I asked it to, similar to the article's author. However, it kept getting it wrong and then started to repeat previous incorrect answers. I finally said "you repeated an incorrect answer from before" and then i…

I kinda like its method though. Think I'm just gonna throw my own "Internal server error" response out when I get the 12th frustrating email reply and I've had enough.

Re: ChatGPT, Rot13, and Daniel Kahneman

#14

Post by joshuahedlund »

This is a really clear explanation of what’s happening in when someone says “it’s not thinking it’s just pattern-matching” and someone else says “well isn’t that all humans really do too?”

Rather: ChatGPT can engage in some level of System 1 thinking, by pattern-matching and even cleverly recombining the entire corpus of System 1 thinking displayed all over the internet. Humans do engage in this type of thinking and it’s a significant accomplishment for an AI. But humans also engage in System 2 thinking. My bet is AGI requires System 2. It’s not clear if that is a gap of degree or kind for this type of AI.

Re: ChatGPT, Rot13, and Daniel Kahneman

#15

Post by knaik94 »

I was playing around with a similar kind of problem trying to get it to decode Caesar cipher encoded text. I asked it to start by doing a frequency analysis of the ciphertext and for the most part it was right, but counted an extra instance of a letter. From there I tried making it loop through different shift values and made the stop condition finding a real word.

It was able to shift by a constant number successfully and even tried shifting both forward (+2) and backward (-2) looking for valid words without additional prompting. But it did not loop through every possibility and stopped having found a word that wasn't real. The interesting thing was that asking the model if the word it found was real with a follow-up question, it correctly identified that it gave an incorrect answer.

Part of why it failed to find a word is that it did an incorrect step going from EXXEG... to TAAAT... as a poor attempt of applying the frequency analysis. It understood that E shouldn't substitute with E and moved on to E->T, but the actual substitution failed.

The limitations of context memory and error checking are interesting and not something I expected from this model. The unprompted test of both positive and negative shift values shows some sort of system 2 thinking, but it's doesn't seem consistent.

https://twitter.com/Knaikk/status/1600001061971849216

Re: ChatGPT, Rot13, and Daniel Kahneman

#16

Post by PostOnce »

It is as though its mathematical abilities are incomplete in their training, and wildly, incomprehensibly convoluted:

I tried many base64 strings and they all decoded correctly until:

It "decoded" the base64 string for "which actress is the best?" except that it replaced "actress" with "address"... there is no off-by-one error that brings you to that.

You may try 100 base64 strings and they all decode correctly... only to find, in fact, that it DOES NOT know how to decode base64 reliably.

This tool could be a 50x accelerator for an expert, but absolutely ruinous to a non-expert in any given field.

I also got it to draw an icosahedron whose points were correct but whose triangles were draw incorrectly, so if I create a convex hull over it, it's correct.

The kinds of mistakes it makes are so close but so far at the same time. It sometimes writes complete working programs that are off by a single variable assignment, or sometimes they're just perfect, other times, they're nonsensical and call magic pseudocode functions or misunderstand the appropriate algorithm for a context (e.g. audio vs text compression).

It can provide citations for legal opinions -- but decades old citations that don't reflect current precedent.

God help us all if they plug it into some robot arms or give it the ability to run arbitrary code it outputs on a network interface.

Let's say they dump another 10 billion dollars into it and dectuple the size of the network, will it suddenly become legitimately capable, and not just "wow that's close" but actually startlingly competent in many more fields?

I could see this thing causing a war by all manner of means, whether its putting many out of work, making beguiling suggestions, outputting dangerous code, or, I'm sure, a million things that don't spring immediately to my small mind.

Re: ChatGPT, Rot13, and Daniel Kahneman

#17

Post by ftufek »

volleygman180 wrote: post #5

I love how it eventually gives up and throws an "Internal Server error". Over the weekend, I was finding ChatGPT giving me incorrect answers for some things too. In one case, it would try to correct itself when I asked it to, similar to the article's author. However, it kept getting it wrong and then started to repeat previous incorrect answers. I finally said "you repeated an incorrect answer from before" and then i…

I believe the internal server error is because of server load, unrelated to the query itself. I've been using chatgpt since it came out, as it got more viral, it started becoming slower and slower and now, it just randomly gives server errors, hopefully it'll be solved as they scale their systems.

Re: ChatGPT, Rot13, and Daniel Kahneman

#19

Post by yunyu »

There's a way simpler answer than this Type I Type II thinking stuff. Most LLMs like GPT are not trained on the level of individual characters – they process input and outputs on the level of subword units that compose multiple characters to support long context windows (i.e. "door" instead of "d", "o", "o", "r"). As a result, they do poorly on character manipulation tasks. You can get some insight here: https://beta.openai.com/tokenizer

This is a solved problem with models trained on byte-level objectives without tokenization like ByT5 (if you tried this task on one of those, it would probably work perfectly with a few samples). In GPT’s case, there’s a trade off between having a long context window vs being good at character level tasks, and OpenAI picked the former.

Post reply on HN