Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Teaching ChatGPT to speak my son’s invented language

szopa.medium.com

Post reply on HN
Print view View original

51–60 of 212 posts

Re: Teaching ChatGPT to speak my son’s invented language

#51

Post by skybrian »

famouswaffles wrote: post #37
skybrian wrote: post #29

Earlier quoted context omitted.

I agree that it’s a neat demo, but it’s not all that useful in itself. You could also do this by writing a function in a programming language (if it weren’t built in), to arbitrary accuracy, and it doesn’t cost anything to run. A more practical thing to do for algorithms is probably to use ChatGPT to help you write the function you need.

The significance of the paper is moreso the implications of how far ICL can take you rather than the ease/viability of the solution proposed. Sure there are better methods for arithmetic but arithmetic is extremely quantifiable with rigid steps. What happens when you step out of that kind of domain ? Like the above blog. or Code documentation. For example, you can paste new documentation in a gpt-4 instance and it wi…

You've got a limited context window (for now). There's only so much you can put into a prompt, so how much you can teach it this way is going to be pretty limited. Whatever you teach it had better be the primary task you're using it for.

You can't do it for everything, but if you can generate code and run it outside the LLM, you should.

Re: Teaching ChatGPT to speak my son’s invented language

#52

Post by famouswaffles »

localplume wrote: post #40
chankstein38 wrote: post #27

Earlier quoted context omitted.

You're trying the old primary school method of testing things based on the wrong criteria. Why does it need to play tic-tac-toe, hangman, or rock paper scissors? Why do you think a language model would be good at those things? Similarly, why would you expect a thing trained on the context of text would be good at reversing strings? It's amazing it's as good as it is at these things because it doesn't really make sens…

Because those games are just a way to measure how an internal state changes with moves initiated by the ego and initiated by someone else. The point is that there is no consistant internal state because it hallucinates and spotaneously changes. its like telling the language model a story, and getting it to repeat certain facts about it or you make additions to the story. its the exact same thing. It needs quite a lot…

It can play tic-tac toe, chess just fine

https://pastebin.com/cPwpZnZu

https://twitter.com/zswitten/status/1631107663500304384

Re: Teaching ChatGPT to speak my son’s invented language

#55

Post by Baeocystin »

knome wrote: post #24
afro88 wrote: post #10

Earlier quoted context omitted.

Both GPT-3.5 and GPT-4 versions of ChatGPT are limited to 4k tokens, even though GPT-4 is capable of 32k. This leads me to believe that part of the reason for some of the mediocre results OP saw was because they hit the token limit and ChatGPT started "forgetting" earlier parts of the conversation.

GPT-4 allows you to use 8k of context in their current beta, if you're using the chat api directly. It will be interesting ( and probably expensive, lol ) when they open it to a full 32k.

I'm really looking forward to being able to use a personalized LoRa on top of a GPT-4+ class model. I want to be able to train on all of may writing over the past few decades and interrogate the history of my ideas, and I think this would be tremendously valuable for writers of all kinds. Heck, think of the value of training (with their blessing) on something like /r/AskHistorians, or other deep-dive, high quality fora.

Re: Teaching ChatGPT to speak my son’s invented language

#56

Post by tel »

kloch wrote: post #33
famouswaffles wrote: post #9

In context learning is hands down the biggest breakthrough of LLMs. The flexibility the model displays without updating weights is genuinely mind blowing, bordering on absurd especially if you've trained other kinds of models before. See here - https://imgur.com/a/w3DAYOi from the paper - https://arxiv.org/abs/2211.09066 GPT 3.5's (4 is much much better) addition accuracy tanks after 2 digits. However, by approaching…

> you can supercharge accuracy to basically 100% for up to 13 digit addition and >90% after. Is the ~13 digit limitation due to the model itself (how/how well it was trained) or simply the use of double precision in the model weights (which maxes out around 15 digits of precision).

In order for it to be the second you'd need to assume that at least some part of the critical reasoning of the LLM involves storing the data of the number in a single model activation. This is pretty unlikely, as models tend to store information across many activations simultaneously. I don't know this for a fact, you'd need to do brain surgery on GPT-4 and it'd be hard even in that case, but most studies of ANN processing would suggest that the null hypothesis is to assume that the information is widely distributed and not sigfig limited in that way.

Re: Teaching ChatGPT to speak my son’s invented language

#57

Post by robga »

I am curious if the advent of GPT and LLMs allows linguistic theorists to adjudicate where we are with understanding the language instinct and settling the Chomsky vs Pinker vs Others debate.

Perhaps it is entirely irrelevant as GLT has learned through billions of examples a child never could. Or perhaps it is totally relevant as it can synthesise billions of examples better than any linguist.

Re: Teaching ChatGPT to speak my son’s invented language

#58

Post by Buttons840 »

famouswaffles wrote: post #9

In context learning is hands down the biggest breakthrough of LLMs. The flexibility the model displays without updating weights is genuinely mind blowing, bordering on absurd especially if you've trained other kinds of models before. See here - https://imgur.com/a/w3DAYOi from the paper - https://arxiv.org/abs/2211.09066 GPT 3.5's (4 is much much better) addition accuracy tanks after 2 digits. However, by approaching…

Being able to learn within context, without updating weights is amazing. Imagine how much more efficient and/or powerful it could be if we found a way to update the weights in real time.

Re: Teaching ChatGPT to speak my son’s invented language

#59

Post by TeMPOraL »

dfxm12 wrote: post #26

Did it actually speak the language or did it just translate text? I'm not trying to be pedantic; these are two very different tasks.

It could not speak because it has no mouth, but as far as the translation go, I'd say somewhere in between. AFAIU, there's been some indication that GPT-4 works with concepts (so e.g. if it gets extra training for a specific task in one language, its performance on that task improves in other languages as well), GPT-3.5 probably does too, to a lesser extent.

Re: Teaching ChatGPT to speak my son’s invented language

#60

Post by thebestgamers »

skybrian wrote: post #51
famouswaffles wrote: post #37

Earlier quoted context omitted.

The significance of the paper is moreso the implications of how far ICL can take you rather than the ease/viability of the solution proposed. Sure there are better methods for arithmetic but arithmetic is extremely quantifiable with rigid steps. What happens when you step out of that kind of domain ? Like the above blog. or Code documentation. For example, you can paste new documentation in a gpt-4 instance and it wi…

You've got a limited context window (for now). There's only so much you can put into a prompt, so how much you can teach it this way is going to be pretty limited. Whatever you teach it had better be the primary task you're using it for. You can't do it for everything, but if you can generate code and run it outside the LLM, you should.

[flagged]
Post reply on HN