Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT, Rot13, and Daniel Kahneman

jameswillia.ms

Post reply on HN
Print view View original

101–110 of 213 posts

Re: ChatGPT, Rot13, and Daniel Kahneman

#101

Post by Pxtl »

notpushkin wrote: post #82
uh_uh wrote: post #50

Earlier quoted context omitted.

This doesn't really explain it, because the following prompt works flawlessly: "Print each letter of the word "Artificial intelligence" on a new line. Each line should contain a single letter." But the following doesn't: "Print the ROT13 encoding of each letter of the word "Artificial intelligence" on a new line. Each line should contain a single letter." So it gets hung up specifically on ROT13.

I've tried this. Prompt: Print each letter of the word "Artificial intelligence" on a new line. Each line should contain a single letter. ChatGPT: [prints "Artificial intelligence", each letter on its own line] Prompt: Awesome! Now do ROT-13 for each letter. ChatGPT: ROT-13 is a simple substitution cipher that replaces a letter with the letter 13 letters after it in the alphabet. For example, A becomes N and B become…

What's neat is that its made-up words sound englishesque. "Aitically languagement" is wrong in very interesting ways. It's generating chains of rot13 letters that map to common English syllables that are only tangentially related to the letters it worked with.

Re: ChatGPT, Rot13, and Daniel Kahneman

#102

Post by TeMPOraL »

stared wrote: post #31

This "fast and instinctual" is very common for deep learning models. For example, here with a friend, we were showing ConvNets seemingly-NSFW images: https://medium.com/@marekkcichy/does-ai-have-a-dirty-mind-to... (note: ALL photos are nudity-free; yet, I advise not to watch it in your office, as people taking glimpses will think that you watch some adult content; therefore, it is metaphorically SFW, but actually mig…

> Almost always, classifiers are tricked. We are as well... but only at first glance. Afterward, it is evident that these are innocent images.

I recommend reading to the end and pondering the reveal of the mystery of The Lamp.

This is the closest I've ever seen to an image whose NSFW status flips back and forth purely depending on your "System 2" knowledge.

It also highlights we're really tackling automated NSFW detection by going after a proxy, not the real thing - the algorithms try to recognize what is depicted on a given image, whereas the true question to ask is, is that image triggering emotions we don't want our audience to experience (arousal, for porn, but others - like disgust - for different types of NSFW).

But then, I realize, perhaps it's for the better, because if someone builds an image classifier that detects induced emotions, the ad industry will use it to finally destroy everything that's good in life.

Re: ChatGPT, Rot13, and Daniel Kahneman

#103

Post by Telemakhos »

freediver wrote: post #89
tkgally wrote: post #72

I find it amusing that, at present, ChatGPT seems to be lousy at mathematical-type reasoning while being very good at natural language use. That is the opposite of what many people, including me, have come to expect of computers. I have worked for many years in translation, lexicography, and language education, and I am flabbergasted at how well ChatGPT handles natural language. It can produce example sentences of po…

ChatGPT is a natural language model, meaning it has been trained on vast amounts of text and thus is good at processing and outputting text back. To it, numbers follow the rules of language, and not math, unlike for example a dedicated calculator app. Only thanks to seeing numbers in vast amount of text it was trained on, it is able to do common math relatively well, and anything uncommon very poorly.

This will sound trite, but I wonder whether there's a lesson here to be learned about human education. We put a great deal of effort into educating our children in language and math, to the extent that those form the major branches of our standardized testing for college admissions tests (where they are still used). We treat them as separate subjects, perhaps for good reason, as we also do science and geography and art and music. Can ChatGPT's successes and shortcomings give us insight into how we should or should not educate our own progeny?

Re: ChatGPT, Rot13, and Daniel Kahneman

#104

Post by bilsbie »

yunyu wrote: post #19

There's a way simpler answer than this Type I Type II thinking stuff. Most LLMs like GPT are not trained on the level of individual characters – they process input and outputs on the level of subword units that compose multiple characters to support long context windows (i.e. "door" instead of "d", "o", "o", "r"). As a result, they do poorly on character manipulation tasks. You can get some insight here: https://beta…

So how does it do as good as it did on this task? It ended up getting some of the words.

Re: ChatGPT, Rot13, and Daniel Kahneman

#105

Post by TeMPOraL »

teovall wrote: post #95
teovall wrote: post #88

It's able to do base64 encoding decoding pretty well, at least if you tell it to simulate a Linux terminal first. https://i.imgur.com/MHxjVHy.png I want you to act as a Linux terminal. I will type commands and you will reply with what the terminal should show. I want you to only reply with the terminal output inside one unique code block, and nothing else. Do not write explanations. Do not type commands unless I inst…

After some more testing, you can even enter base64 encoded text directly as an input and if you do, it usually base64 encodes its response. V2h5IGRpZCB0aGUgY2hpY2tlbiBjcm9zcyB0aGUgcm9hZD8= SXQgaXMgbm90IGNsZWFyIHdoYXQgeW91IGFyZSBhc2tpbmcgYWJvdXQsIGNvdWxkIHlvdSBwbGVhc2UgcmVnaXN0ZXIgdGhlIHF1ZXN0aW9uIGluIGEgbW9yZSBjbGVhciBhbmQgZGlzY3Vzc2libGUgd2F5PyBJIGNhbiBoZWxwIHlvdSB3aXRoIGFueSBxdWVzdGlvbnMgeW91IG1pZ2h0IGhhdmUgYWJvdXQ…

That's insane! Particularly compared to all those other failed attempts elsewhere in the thread.

Makes me wonder, is anyone keeping a unit test suite for all this stuff? Between inherent[0] randomness in the model, and OpenAI team constantly tweaking it[1] to close gaps people use to make it produce undesirable content, techniques like the one you discovered will break sooner or later - it would be great to know when that happens, and perhaps over time, figure out some robust ones.

(OTOH, there's a limit to what one can learn from this - eventually, they'll drop another model, with its own prompt idiosyncrasies. I'm still bewildered people talk about "prompt engineering" as if it was a serious discipline or occupation, given that it's all just tuning your phrasing to transient patterns in the model that disappear just as fast as they're discovered.)

--

[0] - From the user interface side; the model underneath is probably deterministic.

[1] - If one is to believe the anecdotes here and on Reddit, it would seem many such "prompt hacks" have a shelf life of few hours to a day, before they stop working, presumably through OpenAI intervention.

Re: ChatGPT, Rot13, and Daniel Kahneman

#106

Post by abitnegative »

Telemakhos wrote: post #103
freediver wrote: post #89

Earlier quoted context omitted.

ChatGPT is a natural language model, meaning it has been trained on vast amounts of text and thus is good at processing and outputting text back. To it, numbers follow the rules of language, and not math, unlike for example a dedicated calculator app. Only thanks to seeing numbers in vast amount of text it was trained on, it is able to do common math relatively well, and anything uncommon very poorly.

This will sound trite, but I wonder whether there's a lesson here to be learned about human education. We put a great deal of effort into educating our children in language and math, to the extent that those form the major branches of our standardized testing for college admissions tests (where they are still used). We treat them as separate subjects, perhaps for good reason, as we also do science and geography and a…

Are you asking if we should just teach kids language and hope that they learn math, science, geography, art, and music as a side effect?

Re: ChatGPT, Rot13, and Daniel Kahneman

#108

Post by tkgally »

Telemakhos wrote: post #103
freediver wrote: post #89

Earlier quoted context omitted.

ChatGPT is a natural language model, meaning it has been trained on vast amounts of text and thus is good at processing and outputting text back. To it, numbers follow the rules of language, and not math, unlike for example a dedicated calculator app. Only thanks to seeing numbers in vast amount of text it was trained on, it is able to do common math relatively well, and anything uncommon very poorly.

This will sound trite, but I wonder whether there's a lesson here to be learned about human education. We put a great deal of effort into educating our children in language and math, to the extent that those form the major branches of our standardized testing for college admissions tests (where they are still used). We treat them as separate subjects, perhaps for good reason, as we also do science and geography and a…

That doesn’t sound trite to me at all. ChatGTP raises major issues for the future of education in many areas, and discussions about how to deal with them have just begun.

Re: ChatGPT, Rot13, and Daniel Kahneman

#109

Post by constantcrying »

zarzavat wrote: post #93
freediver wrote: post #89

Earlier quoted context omitted.

ChatGPT is a natural language model, meaning it has been trained on vast amounts of text and thus is good at processing and outputting text back. To it, numbers follow the rules of language, and not math, unlike for example a dedicated calculator app. Only thanks to seeing numbers in vast amount of text it was trained on, it is able to do common math relatively well, and anything uncommon very poorly.

As pointed out by Yannic[0], ChatGPT is actually a source code model first, then they trained natural language model on top of that. Source code is still language but it has more math in it. But the truth is we don’t know. I personally wouldn’t be surprised if they do train it on a whole bunch of calculator output to boost its numerical reasoning. [0] https://youtu.be/0A8ljAkdFtg 7:21

The whole approach of a language model makes it hard for it to understand mathematical operations. It generates probabalistically the next text segment that it thinks "fits". That approach does not work for mathematics and "learning" mathematical operations seems to be out if scope for language models, but that is somewhat of an inherent problem.

Re: ChatGPT, Rot13, and Daniel Kahneman

#110

Post by jaybrendansmith »

yunyu wrote: post #19

There's a way simpler answer than this Type I Type II thinking stuff. Most LLMs like GPT are not trained on the level of individual characters – they process input and outputs on the level of subword units that compose multiple characters to support long context windows (i.e. "door" instead of "d", "o", "o", "r"). As a result, they do poorly on character manipulation tasks. You can get some insight here: https://beta…

I think the type I/II thinking does make sense. I tried the classic baseball bat riddle, which is a great riddle because type I thinking fails: A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? GPT chat answered: The bat costs $1.00 + $0.10 = $>1.10. The ball costs $1.10 - $1.00 = $>0.10. Answer: \boxed{0.10}.
Post reply on HN