Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Curious about the training data of OpenAI's new GPT-OSS models? I was too

twitter.com

Post reply on HN
Print view View original

21–30 of 62 posts

Re: Curious about the training data of OpenAI's new GPT-OSS models? I was too

#21

Post by meowface »

esperent wrote: post #17

> the chains start in English but slowly descend into Neuralese What is Nueralese? I tried searching for a definition but it just turns up a bunch of Less Wrong and Medium articles that don't explain anything. Is it a technical term?

It's a term somewhat popularized by the LessWrong/rationalism community to refer to communication (self-communication/note-taking/state-tracking/reasoning, or model-to-model communication) via abstract latent space information rather than written human language. Vectors instead of words.

One implication leading to its popularity by LessWrong is the worry that malicious AI agents might hide bad intent and actions by communicating in a dense, indecipherable way while presenting only normal intent and actions in their natural language output.

Re: Curious about the training data of OpenAI's new GPT-OSS models? I was too

#22

Post by nopinsight »

esperent wrote: post #17

> the chains start in English but slowly descend into Neuralese What is Nueralese? I tried searching for a definition but it just turns up a bunch of Less Wrong and Medium articles that don't explain anything. Is it a technical term?

The author might use it as an analogy to mentalese but for neural networks.

https://en.wiktionary.org/wiki/mentalese

EDIT: After reading the original thread in more detail, I think some of the sibling comments are more accurate. In this case, neuralese is more like language of communication expressed by neural networks, rather than its internal representation.

Re: Curious about the training data of OpenAI's new GPT-OSS models? I was too

#23

Post by AmazingTurtle »

james-bcn wrote: post #19

This looks very interesting but I don't really understand what he has done here. Can someone explain the process he has gone through in this analysis?

He presented an empty prompt to gpt OSS and let it run many times. Through temperature, the results vary quite a lot. He sampled the results.

Feeding an empty prompt to a model can be quite revealing on what data it was trained on

Re: Curious about the training data of OpenAI's new GPT-OSS models? I was too

#25

Post by verisimi »

meowface wrote: post #21
esperent wrote: post #17

> the chains start in English but slowly descend into Neuralese What is Nueralese? I tried searching for a definition but it just turns up a bunch of Less Wrong and Medium articles that don't explain anything. Is it a technical term?

It's a term somewhat popularized by the LessWrong/rationalism community to refer to communication (self-communication/note-taking/state-tracking/reasoning, or model-to-model communication) via abstract latent space information rather than written human language. Vectors instead of words. One implication leading to its popularity by LessWrong is the worry that malicious AI agents might hide bad intent and actions by c…

> malicious AI agents might hide bad intent and actions by communicating in a dense, indecipherable way while presenting only normal intent and actions in their natural language output.

you could edit this slightly to extract a pretty decent rule for governance, like so:

> malicious agents might hide bad intent and actions by communicating in a dense, indecipherable way while presenting only normal intent and actions in a natural way

It applies to ai, but also many other circumstances where the intention is that you are governed - eg medical, legal, financial.

Thanks!

Re: Curious about the training data of OpenAI's new GPT-OSS models? I was too

#26

Post by randomNumber7 »

k310 wrote: post #10
Epskampie wrote: post #9

Earlier quoted context omitted.

https://xcancel.com/jxmnop/status/1953899426075816164

Thanks! I've seen a lot of stuff come and go, so thanks for the reminder. For example, Libgen is out of commission, and the substitutes are hell to use. Summary of what's up and not up: https://open-slum.org/

> Libgen is out of commission, and the substitutes are hell to use

Somehow I also preferred libgen, but I don't think annas archive is "hell to use".

Re: Curious about the training data of OpenAI's new GPT-OSS models? I was too

#27

Post by Mistletoe »

Hard_Space wrote: post #14
puttycat wrote: post #13

> OpenAI has figured out RL. the models no longer speak english What does this mean?

Interesting. This happens in Colossus: The Forbin Project (1970), where the rogue AI escapes the semantic drudgery of English and invents its own compressed language with which to talk to its Russian counterpart.

It also happens in Ex Machina at the end when the two androids whisper and talk to each other in their special faster language. I always found this to be one of the most believable, real things from that movie and one of my favorite parts.

Re: Curious about the training data of OpenAI's new GPT-OSS models? I was too

#28

Post by jdfr »

OP seems to have run a programming language detector on the generated texts, and made a graph of programming language frecuencies: https://pbs.twimg.com/media/Gx2kvNxXEAAkBO0.jpg?name=orig

As a result, OP seems to think the model was trained on a lot of Perl: https://xcancel.com/jxmnop/status/1953899440315527273#m

LOL! I think these results speak more to the flexibility of Perl than any actual insight on the training data! After all, 93% of inkblots are valid Perl scripts: https://www.mcmillen.dev/sigbovik/

Re: Curious about the training data of OpenAI's new GPT-OSS models? I was too

#29

Post by YeGoblynQueenne »

AmazingTurtle wrote: post #23
james-bcn wrote: post #19

This looks very interesting but I don't really understand what he has done here. Can someone explain the process he has gone through in this analysis?

He presented an empty prompt to gpt OSS and let it run many times. Through temperature, the results vary quite a lot. He sampled the results. Feeding an empty prompt to a model can be quite revealing on what data it was trained on

Not an empty prompt but a one-token prompt:

>> i sample tokens based on average frequency and prompt with 1 token

https://x.com/iamgrigorev/status/1953919577076683131

Re: Curious about the training data of OpenAI's new GPT-OSS models? I was too

#30

Post by johnisgood »

jdfr wrote: post #28

OP seems to have run a programming language detector on the generated texts, and made a graph of programming language frecuencies: https://pbs.twimg.com/media/Gx2kvNxXEAAkBO0.jpg?name=orig As a result, OP seems to think the model was trained on a lot of Perl: https://xcancel.com/jxmnop/status/1953899440315527273#m LOL! I think these results speak more to the flexibility of Perl than any actual insight on the training…

That inkblot thing can be created for any language.
Post reply on HN