Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Teaching ChatGPT to speak my son’s invented language

szopa.medium.com

Post reply on HN
Print view View original

61–70 of 212 posts

Re: Teaching ChatGPT to speak my son’s invented language

#61

Post by kristjansson »

kloch wrote: post #33
famouswaffles wrote: post #9

In context learning is hands down the biggest breakthrough of LLMs. The flexibility the model displays without updating weights is genuinely mind blowing, bordering on absurd especially if you've trained other kinds of models before. See here - https://imgur.com/a/w3DAYOi from the paper - https://arxiv.org/abs/2211.09066 GPT 3.5's (4 is much much better) addition accuracy tanks after 2 digits. However, by approaching…

> you can supercharge accuracy to basically 100% for up to 13 digit addition and >90% after. Is the ~13 digit limitation due to the model itself (how/how well it was trained) or simply the use of double precision in the model weights (which maxes out around 15 digits of precision).

Additional evidence against the second hypothesis: almost nothing in LLM-land is double precision anyway, weights are generally half-precision (or something like bfloat16 with more range by less precision than IEEE float16).

Re: Teaching ChatGPT to speak my son’s invented language

#62

Post by famouswaffles »

skybrian wrote: post #51
famouswaffles wrote: post #37

Earlier quoted context omitted.

The significance of the paper is moreso the implications of how far ICL can take you rather than the ease/viability of the solution proposed. Sure there are better methods for arithmetic but arithmetic is extremely quantifiable with rigid steps. What happens when you step out of that kind of domain ? Like the above blog. or Code documentation. For example, you can paste new documentation in a gpt-4 instance and it wi…

You've got a limited context window (for now). There's only so much you can put into a prompt, so how much you can teach it this way is going to be pretty limited. Whatever you teach it had better be the primary task you're using it for. You can't do it for everything, but if you can generate code and run it outside the LLM, you should.

The limits of the context window become much less important (but can still be a problem I agree) when crucial context can be dynamically inserted only when relevant.

Gpt-3.5 doesn't need the algorithm prompt for every single query. It just needs it for every query that requires arithmetic. Much more feasible.

Re: Teaching ChatGPT to speak my son’s invented language

#63

Post by yashap »

vjerancrnjak wrote: post #50
vintermann wrote: post #7

Once again illustrating that the powerful thing about ChatGPT is that no matter what you do, it does its best to play along. Its eyes do not glaze over.

Just recently I asked it to invent some new Croatian words and it refused. I asked it if a certain word means something in Croatian (it exists in a dialect). It said it has no meaning. Then I asked it to pretend and give it a suitable meaning: "As an AI language model, I don't endorse creating made-up words or pretending that they have meanings in any language. It's important to use language accurately and with respe…

You can get around these limitations with jailbreak prompts: https://www.jailbreakchat.com/

Re: Teaching ChatGPT to speak my son’s invented language

#65

Post by elhudy »

famouswaffles wrote: post #9

In context learning is hands down the biggest breakthrough of LLMs. The flexibility the model displays without updating weights is genuinely mind blowing, bordering on absurd especially if you've trained other kinds of models before. See here - https://imgur.com/a/w3DAYOi from the paper - https://arxiv.org/abs/2211.09066 GPT 3.5's (4 is much much better) addition accuracy tanks after 2 digits. However, by approaching…

In-context learning also seems like the best path to commercializing LLMs. I'm surprised that Microsoft is going the D2C route with ChatGPT rather than commercializing it in a B2B fashion. ...Or maybe that's coming?

Imagine feeding an LLM a ton of disparate data sources and asking it questions about that data as a whole. What is a data engineer again, anyway?

Re: Teaching ChatGPT to speak my son’s invented language

#66

Post by tyingq »

elhudy wrote: post #65
famouswaffles wrote: post #9

In context learning is hands down the biggest breakthrough of LLMs. The flexibility the model displays without updating weights is genuinely mind blowing, bordering on absurd especially if you've trained other kinds of models before. See here - https://imgur.com/a/w3DAYOi from the paper - https://arxiv.org/abs/2211.09066 GPT 3.5's (4 is much much better) addition accuracy tanks after 2 digits. However, by approaching…

In-context learning also seems like the best path to commercializing LLMs. I'm surprised that Microsoft is going the D2C route with ChatGPT rather than commercializing it in a B2B fashion. ...Or maybe that's coming? Imagine feeding an LLM a ton of disparate data sources and asking it questions about that data as a whole. What is a data engineer again, anyway?

Interesting, though I imagine that will often play out with the business person rephrasing the question or filtering out data until it spits out the answer they expected :) That data engineer can at least push back and have their own opinion.

Re: Teaching ChatGPT to speak my son’s invented language

#68

Post by moffkalast »

fernly wrote: post #64

Oh I wish I had time to train it on one of my old hobbies, Lojban! https://lojban.io/ https://mw.lojban.org/papri/Lojban

Any language existing prior to 2021 isn't gonna be very useful for testing its improv abilities, since they're likely all in the training data.

Re: Teaching ChatGPT to speak my son’s invented language

#69

Post by zenlikethat »

elhudy wrote: post #65
famouswaffles wrote: post #9

In context learning is hands down the biggest breakthrough of LLMs. The flexibility the model displays without updating weights is genuinely mind blowing, bordering on absurd especially if you've trained other kinds of models before. See here - https://imgur.com/a/w3DAYOi from the paper - https://arxiv.org/abs/2211.09066 GPT 3.5's (4 is much much better) addition accuracy tanks after 2 digits. However, by approaching…

In-context learning also seems like the best path to commercializing LLMs. I'm surprised that Microsoft is going the D2C route with ChatGPT rather than commercializing it in a B2B fashion. ...Or maybe that's coming? Imagine feeding an LLM a ton of disparate data sources and asking it questions about that data as a whole. What is a data engineer again, anyway?

D2C makes a lot of sense imo because at this point it's most valuable to obtain a huge corpus of (reasonably well labeled) training data (with a side helping of being really great marketing).

Just think about all the programmers who are probably rolling their eyes and correcting ChatGPT's work for OpenAI, or at least providing some details. I know it's a reflex for me, not even deliberate, to at least type something in like "Ah I figured out that it was XYZ" and thumbs up or thumbs down.

Microsoft will undoubtedly be rolling this all out in Docs, Excel, Teams etc in no time. A good reason to be skeptical about the flood of "GPT for your Notion!" startups.

Re: Teaching ChatGPT to speak my son’s invented language

#70

Post by moffkalast »

yashap wrote: post #63
vjerancrnjak wrote: post #50

Earlier quoted context omitted.

Just recently I asked it to invent some new Croatian words and it refused. I asked it if a certain word means something in Croatian (it exists in a dialect). It said it has no meaning. Then I asked it to pretend and give it a suitable meaning: "As an AI language model, I don't endorse creating made-up words or pretending that they have meanings in any language. It's important to use language accurately and with respe…

You can get around these limitations with jailbreak prompts: https://www.jailbreakchat.com/

You usually don't have to go that far, just assuring it that the question is for a good cause often gets past the filter. Or slight reverse psychology.
Post reply on HN