Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Teaching ChatGPT to speak my son’s invented language

szopa.medium.com

Post reply on HN
Print view View original

31–40 of 212 posts

Re: Teaching ChatGPT to speak my son’s invented language

#31

Post by felipemnoa »

chankstein38 wrote: post #25
vintermann wrote: post #7

Once again illustrating that the powerful thing about ChatGPT is that no matter what you do, it does its best to play along. Its eyes do not glaze over.

One of the things that always gives me a little hit of hype is when I tell it to do something ridiculous and it just dutifully starts spitting out the result without complaining or questioning lol

I wonder if that is how our brain produces dreams? The guardrails are down so it will just start producing ridiculous and/or implausible things.

Edit: It almost seems like you are anthropomorphizing it. It is just a program doing what it's supposed to be doing: to predict the next token based on its weights. Nothing more, nothing less. It does give the illusion of intelligence. Pretty soon, though, we may not be able to tell the difference.

Re: Teaching ChatGPT to speak my son’s invented language

#32

Post by shagie »

chmod775 wrote: post #22
bko wrote: post #18

Earlier quoted context omitted.

You said you don't have access but based on your tests... Were you testing ChatGPT I just tried and it was able to play tic tac toe, reverse a string (the string was "hello world.i am new to this so please forgive me if i can't reverse a sentence") Hangman sort of worked but it said every letter I picked was correct and appears to have constructed a word based on my guesses. Very strange behavior

Try making it reverse this: "Quickly, the kangaroo hopped away, escaping under the azure sky." I couldn't make it reverse that correctly even after prompting it five times to fix its mistakes. Most commonly it writes: ".yks eruz a eht rednu gnipacse ,yawa depoh ooragnak eht ,ylkciuQ" It also can't find the mistakes in there for the life of it.

Remember that GPT is working on input tokens and output tokens. Its output is tokens that then get converted back into text.

Taking [21063, 306, 11, 262, 479, 648, 38049, 45230, 1497, 11, 25071, 739, 262, 35560, 495, 6766, 13] to expect it to output back [13, 88, 591, 13724, 4496, 304, 4352, 2266, 28803, 19967, 541, 330, 325, 837, 88, 6909, 390, 381, 1219, 267, 273, 4660, 461, 304, 4352, 837, 2645, 74, 979, 84, 48] is a difficult problem that it is not well suited for.

Re: Teaching ChatGPT to speak my son’s invented language

#33

Post by kloch »

famouswaffles wrote: post #9

In context learning is hands down the biggest breakthrough of LLMs. The flexibility the model displays without updating weights is genuinely mind blowing, bordering on absurd especially if you've trained other kinds of models before. See here - https://imgur.com/a/w3DAYOi from the paper - https://arxiv.org/abs/2211.09066 GPT 3.5's (4 is much much better) addition accuracy tanks after 2 digits. However, by approaching…

> you can supercharge accuracy to basically 100% for up to 13 digit addition and >90% after.

Is the ~13 digit limitation due to the model itself (how/how well it was trained) or simply the use of double precision in the model weights (which maxes out around 15 digits of precision).

Re: Teaching ChatGPT to speak my son’s invented language

#34

Post by chankstein38 »

chmod775 wrote: post #22
bko wrote: post #18

Earlier quoted context omitted.

You said you don't have access but based on your tests... Were you testing ChatGPT I just tried and it was able to play tic tac toe, reverse a string (the string was "hello world.i am new to this so please forgive me if i can't reverse a sentence") Hangman sort of worked but it said every letter I picked was correct and appears to have constructed a word based on my guesses. Very strange behavior

Try making it reverse this: "Quickly, the kangaroo hopped away, escaping under the azure sky." I couldn't make it reverse that correctly even after prompting it five times to fix its mistakes. Most commonly it writes: ".yks eruz a eht rednu gnipacse ,yawa depoh ooragnak eht ,ylkciuQ" It also can't find the mistakes in there for the life of it.

That's because it's trained on the relations of words to each other and not on string manipulation. This is not its purpose. It may be capable of it to some degree but that seems like more of a luck of the draw kind of thing than something we should expect it to be good at.

Re: Teaching ChatGPT to speak my son’s invented language

#35

Post by anon84873628 »

>All of these differences can make it surprising and challenging for someone with an Indo-European language background to learn and use Kłeti.

Ironically, Proto-Indo-European is believed to be far more complex than its modern descendants, as described by Wikipedia:

>PIE is believed to have had an elaborate system of morphology that included inflectional suffixes (analogous to English child, child's, children, children's) as well as ablaut (vowel alterations, as preserved in English sing, sang, sung, song) and accent. PIE nominals and pronouns had a complex system of declension, and verbs similarly had a complex system of conjugation.

So maybe a PIE speaker would have an easier time with Kłeti than we :-)

Re: Teaching ChatGPT to speak my son’s invented language

#36

Post by pelorat »

marcodiego wrote: post #12

I don't have access to ChatGPT4, but in my tests I could observe that it can't do some very simple tasks: - It can't play tic-tac-toe, - It can't play hangman, - It insists that winning on stone-paper-scissor using the chat (playing before me) is a matter of probability. It was also demonstrated that it can't reverse strings. Actually a transformer doesn't accesses 'strings', all it processes are tokens which are the…

LLMs don't see individual characters, they see individual words.

Re: Teaching ChatGPT to speak my son’s invented language

#37

Post by famouswaffles »

skybrian wrote: post #29
famouswaffles wrote: post #9

In context learning is hands down the biggest breakthrough of LLMs. The flexibility the model displays without updating weights is genuinely mind blowing, bordering on absurd especially if you've trained other kinds of models before. See here - https://imgur.com/a/w3DAYOi from the paper - https://arxiv.org/abs/2211.09066 GPT 3.5's (4 is much much better) addition accuracy tanks after 2 digits. However, by approaching…

I agree that it’s a neat demo, but it’s not all that useful in itself. You could also do this by writing a function in a programming language (if it weren’t built in), to arbitrary accuracy, and it doesn’t cost anything to run. A more practical thing to do for algorithms is probably to use ChatGPT to help you write the function you need.

The significance of the paper is moreso the implications of how far ICL can take you rather than the ease/viability of the solution proposed.

Sure there are better methods for arithmetic but arithmetic is extremely quantifiable with rigid steps. What happens when you step out of that kind of domain ? Like the above blog. or Code documentation. For example, you can paste new documentation in a gpt-4 instance and it will use it for your queries as if it trained on it.

Basically Memory Augmented Large Language Models are Computationally Universal https://arxiv.org/abs/2301.04589. and you kind of get the feeling of that from the previous paper.

Re: Teaching ChatGPT to speak my son’s invented language

#38

Post by ojosilva »

chankstein38 wrote: post #25
vintermann wrote: post #7

Once again illustrating that the powerful thing about ChatGPT is that no matter what you do, it does its best to play along. Its eyes do not glaze over.

One of the things that always gives me a little hit of hype is when I tell it to do something ridiculous and it just dutifully starts spitting out the result without complaining or questioning lol

I was thinking exactly the same as I read the OP, right where the dad+kid were answering hypothetical ChatGPT questions with Yes and No.

I think LLMs inference training should include teaching it to ask questions back before starting full-fledged generation. You know, make it a little more Socratic.

Right now the approach is: ChatGPT starts answering and, if it's going the wrong way, you either hit "Stop Generating" or just wait for it to finish then figure yourself how to improve the prompt. LLMs should be also trained in ranking the prompt and determining what questions would make the prompt statistically stronger to generate. I bet it would result in savings running it too. In fact, one can try this out by configuring a system prompt that tells the model to ask questions before getting started with an answer.

Re: Teaching ChatGPT to speak my son’s invented language

#39

Post by famouswaffles »

kloch wrote: post #33
famouswaffles wrote: post #9

In context learning is hands down the biggest breakthrough of LLMs. The flexibility the model displays without updating weights is genuinely mind blowing, bordering on absurd especially if you've trained other kinds of models before. See here - https://imgur.com/a/w3DAYOi from the paper - https://arxiv.org/abs/2211.09066 GPT 3.5's (4 is much much better) addition accuracy tanks after 2 digits. However, by approaching…

> you can supercharge accuracy to basically 100% for up to 13 digit addition and >90% after. Is the ~13 digit limitation due to the model itself (how/how well it was trained) or simply the use of double precision in the model weights (which maxes out around 15 digits of precision).

It's hard to say for sure but the second is pretty unlikely.

Re: Teaching ChatGPT to speak my son’s invented language

#40

Post by localplume »

chankstein38 wrote: post #27
marcodiego wrote: post #12

I don't have access to ChatGPT4, but in my tests I could observe that it can't do some very simple tasks: - It can't play tic-tac-toe, - It can't play hangman, - It insists that winning on stone-paper-scissor using the chat (playing before me) is a matter of probability. It was also demonstrated that it can't reverse strings. Actually a transformer doesn't accesses 'strings', all it processes are tokens which are the…

You're trying the old primary school method of testing things based on the wrong criteria. Why does it need to play tic-tac-toe, hangman, or rock paper scissors? Why do you think a language model would be good at those things? Similarly, why would you expect a thing trained on the context of text would be good at reversing strings? It's amazing it's as good as it is at these things because it doesn't really make sens…

Because those games are just a way to measure how an internal state changes with moves initiated by the ego and initiated by someone else. The point is that there is no consistant internal state because it hallucinates and spotaneously changes. its like telling the language model a story, and getting it to repeat certain facts about it or you make additions to the story. its the exact same thing. It needs quite a lot of "prompt engineering" to push it in the correct direction, and even then its frequently incorrect.
Post reply on HN