Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Teaching ChatGPT to speak my son’s invented language

szopa.medium.com

Post reply on HN
Print view View original

131–140 of 212 posts

Re: Teaching ChatGPT to speak my son’s invented language

#131

Post by FPGAhacker »

famouswaffles wrote: post #118
FPGAhacker wrote: post #115

Earlier quoted context omitted.

I have very limited understanding of chatGPTs architecture, and neural nets in general. From what I've read over the last few days, the "attention" mechanism used in chatGPT and similar LLMs does indeed dynamically change weights of a portion of the model.

the connection between weights are dynamic but the weights themselves aren't being changed i.e being updated in response to the text. when training a model, the forward pass would happen i.e the generation and then depending on how close to truth it was, the configuration settings (aka the weights/neurons) would be adjusted to incorporate whatever little insight was gained from the text. Weights are matrices. The val…

Maybe it comes down to semantics but when I read things like [1] I come away with the idea that the weights are altered. But it could also just be my misunderstanding.

1: https://towardsdatascience.com/an-intuitive-explanation-of-s...

Re: Teaching ChatGPT to speak my son’s invented language

#132

Post by leobg »

yosito wrote: post #116
famouswaffles wrote: post #9

In context learning is hands down the biggest breakthrough of LLMs. The flexibility the model displays without updating weights is genuinely mind blowing, bordering on absurd especially if you've trained other kinds of models before. See here - https://imgur.com/a/w3DAYOi from the paper - https://arxiv.org/abs/2211.09066 GPT 3.5's (4 is much much better) addition accuracy tanks after 2 digits. However, by approaching…

I've noticed if I give chat GPT an algorithm it gives me consistent results when it shows it's work but if I ask it not to show it's work and just give me the answer. It consistently gives me incorrect answers even when I ask it to follow an algorithm that I gave it. I suppose this is similar to humans and probably why my school teachers always told me to show my work, but I'm curious if this has been documented and…

GPT is auto regressive. That means each output token becomes part of the new input sequence. Which is to say, the beginning of the model’s answer becomes part of your prompt.

If the model makes some mistake in the beginning, it now needs to explain / make sense of that mistake.

Kind of like a split-brain patient whom you ask why they got up, and they then say, to get a Coke. [1] In psychology, that is called confabulation. In machine learning, they use “hallucination“, probably so they can use the term across several disciplines, like language, audio, vision, etc.

[1] https://www.brainscape.com/flashcards/chapter-4-hemispheric-...

Re: Teaching ChatGPT to speak my son’s invented language

#133

Post by vintermann »

Vespasian wrote: post #121
yosito wrote: post #116

Earlier quoted context omitted.

I've noticed if I give chat GPT an algorithm it gives me consistent results when it shows it's work but if I ask it not to show it's work and just give me the answer. It consistently gives me incorrect answers even when I ask it to follow an algorithm that I gave it. I suppose this is similar to humans and probably why my school teachers always told me to show my work, but I'm curious if this has been documented and…

LLMs don't have an inner monologue. By their very nature they only "know" what they have written down and must infer the final answer from that token by token. They fundamentally can't do certain things such as complex iteration or track back. When you ask for chain of thought thinking, you allow the LLM to create a "buffer space" and break down the task into more manageable substeps thereby improving the quality of…

The Bing LM, or rather the service, did have "inner monologue" in the sense of text that it would generate, but not show to the user, and treat as "thoughts" to guide the generation of an actual reply that the user would see.

We know this because it happily told us, including the json format it uses internally.

Re: Teaching ChatGPT to speak my son’s invented language

#134

Post by vintermann »

joshribakoff wrote: post #78
Buttons840 wrote: post #58

Earlier quoted context omitted.

Being able to learn within context, without updating weights is amazing. Imagine how much more efficient and/or powerful it could be if we found a way to update the weights in real time.

Maybe even more powerful would be reducing the number of examples needed to learn, eg less than one shot https://www.technologyreview.com/2020/10/16/1010566/ai-machi... updating weights in real time is useless if each update basically does nothing because it takes an insurmountable amount of training, on the other hand if i can give my model a succinct “lesson” i’d then be very willing to wait a while for it to “proc…

Yeah, that's why just updating the weights on the models such as they are doesn't work. But they're right that it's desirable to have some sort of online learning, whether on top of a frozen language model, or through some not yet invented way to do it end to end.

Re: Teaching ChatGPT to speak my son’s invented language

#135

Post by euroderf »

wizzwizz4 wrote: post #82
anon84873628 wrote: post #35

>All of these differences can make it surprising and challenging for someone with an Indo-European language background to learn and use Kłeti. Ironically, Proto-Indo-European is believed to be far more complex than its modern descendants, as described by Wikipedia: >PIE is believed to have had an elaborate system of morphology that included inflectional suffixes (analogous to English child, child's, children, childre…

> as well as ablaut (vowel alterations, as preserved in English sing, sang, sung, song) Interestingly, English is gaining instances of ablaut. For example, dived seems to be being replaced by dove.

plead/pled (driving out pleaded)

Re: Teaching ChatGPT to speak my son’s invented language

#136

Post by Vespasian »

vintermann wrote: post #133
Vespasian wrote: post #121

Earlier quoted context omitted.

LLMs don't have an inner monologue. By their very nature they only "know" what they have written down and must infer the final answer from that token by token. They fundamentally can't do certain things such as complex iteration or track back. When you ask for chain of thought thinking, you allow the LLM to create a "buffer space" and break down the task into more manageable substeps thereby improving the quality of…

The Bing LM, or rather the service, did have "inner monologue" in the sense of text that it would generate, but not show to the user, and treat as "thoughts" to guide the generation of an actual reply that the user would see. We know this because it happily told us, including the json format it uses internally.

Interesting. I didn't know that.

When using gpt-4 directly through the API we can emulate this behavior

Re: Teaching ChatGPT to speak my son’s invented language

#137

Post by DonHopkins »

Prompt: Write a poem in Simlish.

ChatGPT-4:

While it's important to remember that Simlish is a nonsensical language without a proper grammar or vocabulary, I can still attempt to create a playful poem using some known Simlish phrases and sounds. Keep in mind that this poem will not have any real meaning:

    Sul sul! Vadeesh, plurna zib, 
    Badeesh! Dag dag, cribna drib, 
    Froobi flazzle, noogit fraz, 
    Glarbni weeb, frizzle maz.

    Lerpna gort, jibbly zor, 
    Slooble wib, kwebna dor, 
    Skibbi plurt, pleeba narn, 
    Grubble veeb, shazzle darn.

    Wabble gret, minzi glarn, 
    Flartle doo, zibba yarn, 
    Yibbs frazzle, dribna glorp, 
    Vorpti zound, greeble snorp
Enjoy the playful sounds and rhythm of the poem, even though it lacks any real meaning!

Re: Teaching ChatGPT to speak my son’s invented language

#138

Post by DonHopkins »

DonHopkins wrote: post #137

Prompt: Write a poem in Simlish. ChatGPT-4: While it's important to remember that Simlish is a nonsensical language without a proper grammar or vocabulary, I can still attempt to create a playful poem using some known Simlish phrases and sounds. Keep in mind that this poem will not have any real meaning: Sul sul! Vadeesh, plurna zib, Badeesh! Dag dag, cribna drib, Froobi flazzle, noogit fraz, Glarbni weeb, frizzle ma…

Prompt: Write some gangsta rap lyrics in Simlish.

ChatGTP-4:

Creating gangsta rap lyrics in Simlish presents a unique challenge, as Simlish is a nonsensical language, and rap lyrics often rely on meaningful content and wordplay. However, I can attempt to create a playful verse using Simlish phrases and sounds while maintaining a rhythm typical of rap. Keep in mind that these lyrics won't carry any real meaning:

    Verse 1:
    Yo, sul sul, zibba zoob,
    Froodle grubble, wabba woob,
    Glarbni skib, dag dag, dret,
    Cribna drib, yibbs, noogit zet.

    Chorus:
    Gangsta Sim, plurna fraz,
    Vadeesh, plurna, zibba maz,
    Slooble wib, kwebna dor,
    Lerpna gort, jibbly zor.

    Verse 2:
    Flartle doo, minzi glarn,
    Yibbs frazzle, dribna darn,
    Badeesh! Noogit, greeble snorp,
    Vorpti zound, plurt, weeb, glorp.

    Chorus:
    Gangsta Sim, plurna fraz,
    Vadeesh, plurna, zibba maz,
    Slooble wib, kwebna dor,
    Lerpna gort, jibbly zor.
Remember, these lyrics are meant to be playful and fun, as they do not have any real meaning or convey any specific message.

Re: Teaching ChatGPT to speak my son’s invented language

#140

Post by DonHopkins »

DonHopkins wrote: post #138
DonHopkins wrote: post #137

Prompt: Write a poem in Simlish. ChatGPT-4: While it's important to remember that Simlish is a nonsensical language without a proper grammar or vocabulary, I can still attempt to create a playful poem using some known Simlish phrases and sounds. Keep in mind that this poem will not have any real meaning: Sul sul! Vadeesh, plurna zib, Badeesh! Dag dag, cribna drib, Froobi flazzle, noogit fraz, Glarbni weeb, frizzle ma…

Prompt: Write some gangsta rap lyrics in Simlish. ChatGTP-4: Creating gangsta rap lyrics in Simlish presents a unique challenge, as Simlish is a nonsensical language, and rap lyrics often rely on meaningful content and wordplay. However, I can attempt to create a playful verse using Simlish phrases and sounds while maintaining a rhythm typical of rap. Keep in mind that these lyrics won't carry any real meaning: Verse…

Prompt: Write an a cappella cover of a Philip Glass song in Simlish.

ChatGPT-4:

Creating an a cappella cover of a Philip Glass song in Simlish presents an interesting challenge. Philip Glass is known for his minimalist compositions that often consist of repetitive structures, and Simlish is a nonsensical language. Since Glass's compositions are largely instrumental, I will create an a cappella adaptation inspired by his musical style while using Simlish-like sounds.

    Verse 1:
    Doo doo zib, doo doo zib, doo doo zib, doo doo zib,
    Bada bada zoob, bada bada zoob, bada bada zoob, bada bada zoob,
    Glarbni glarbni glarb, glarbni glarbni glarb, glarbni glarbni glarb, glarbni glarbni glarb,
    Sul sul sul, sul sul sul, sul sul sul, sul sul sul.

    Chorus:
    Froodle froodle, zibba zibba,
    Dag dag, drib drib, yibbs yibbs,
    Slooble slooble, wib wib, dor dor,
    Lerpna lerpna, gort gort, zor zor.

    Verse 2:
    Doo doo zib, doo doo zib, doo doo zib, doo doo zib,
    Bada bada zoob, bada bada zoob, bada bada zoob, bada bada zoob,
    Glarbni glarbni glarb, glarbni glarbni glarb, glarbni glarbni glarb, glarbni glarbni glarb,
    Sul sul sul, sul sul sul, sul sul sul, sul sul sul.

    Chorus:
    Froodle froodle, zibba zibba,
    Dag dag, drib drib, yibbs yibbs,
    Slooble slooble, wib wib, dor dor,
    Lerpna lerpna, gort gort, zor zor.
Remember, this adaptation is meant to be playful and fun, as it doesn't have any real meaning or convey any specific message. The focus here is on capturing the essence of Philip Glass's minimalist compositions using Simlish-like sounds in an a cappella context.
Post reply on HN