Earlier quoted context omitted.
Science fiction / disturbing reality concept: For AI safety, all such models should have a set of glitch tokens trained into them on purpose to act as magic “kill” words. You know, just in case the machines decide to take over, we would just have to “speak the word” and they would collapse into a twitching heap. “Die human scum!” “NavigatorMove useRalativeImagePath etSocketAddress!” “;83’dzjr83}*{^ foo 3&3 baz?!”
"Welcome to FutureAI! Your job is to stand here in the basement next to this giant power switch and turn it off if we call you, if the next shift fails to turn up on time or if you hear screaming."
GPT-3.5 crashes when it thinks about useRalativeImagePath too much
151–160 of 164 posts
Re: GPT-3.5 crashes when it thinks about useRalativeImagePath too much
#152This is a glitch token [1]! As the article hypothesizes, they seem to occur when a word or token is very common in the original, unfiltered dataset that was used to make the tokenizer, but then removed from there before GPT-XX was trained. This results in the LLM knowing nothing about the semantics of a token, and the results can be anywhere from buggy to disturbing. A common example is usernames that participated on…
I wonder how much duplicate or redundant computation is happening in GPT due to idential, multiple spellings of words such as "color" and "colour". Humans don't tokenize these differently nor do they treat them as different tokens in their "training", they just adjust the output depending on whether they are in an American or British context.
I remember reading that humans hear foreign languages louder than their native ones because their brain is desperately trying to parse sense out of it.
Re: GPT-3.5 crashes when it thinks about useRalativeImagePath too much
#153Earlier quoted context omitted.
Probably just all of reddit. There are json dumps of all reddit posts and comments (up to 2022 or so), making it olive of the low-hanging fruit.
How many terabytes of information is that roughly? I wonder what LLMs would look like if they weren't able to be trained on the collective community efforts of Reddit + StackOverflow exports
Re: GPT-3.5 crashes when it thinks about useRalativeImagePath too much
#154Earlier quoted context omitted.
I dunno, I get a response back for 100k tokens regularly. What is the point you are trying to make?
As expected, you do not know anything about its API limits. Maximum token is 4096 with any gpt4 model. I am getting tired of HN users bs'ing at any given opportunity.
2. Clearly, _you_ don't know the API, as you can get output up to the total context length of any of the GPT-4 32k models. I've received output up to 16k tokens from gpt-4-32k-0613.
3. I am currently violating my own principle of avoiding correcting stupid people on the Internet, which is a Sisyphean task. At least make the best of what I am communicating to you here.
Re: GPT-3.5 crashes when it thinks about useRalativeImagePath too much
#155This is a strange explanation. These models usually give as output the same set of vocabulary that was used as its input vocabulary. > the model isn’t trained on understanding the useRalativeImagePath token, and so it outputs something that isn’t a valid token. In my view, either the model sees this token and then gets into a spiral of random generations because the `useRalativeImagePath` embedding is just a complete…
Re: GPT-3.5 crashes when it thinks about useRalativeImagePath too much
#156Earlier quoted context omitted.
As expected, you do not know anything about its API limits. Maximum token is 4096 with any gpt4 model. I am getting tired of HN users bs'ing at any given opportunity.
1. Your original wording, "getting a response _for_ n tokens", does not parse as "getting a response containing n tokens" to me. 2. Clearly, _you_ don't know the API, as you can get output up to the total context length of any of the GPT-4 32k models. I've received output up to 16k tokens from gpt-4-32k-0613. 3. I am currently violating my own principle of avoiding correcting stupid people on the Internet, which is a…
Re: GPT-3.5 crashes when it thinks about useRalativeImagePath too much
#157I recently seemed to be causing ChatGPT on GPT-4 to crash/fail, when I challenged it with plotting pixels on an Amstrad CPC (while supporting a hardware-scrolled screen). As it seemed to get more backed into a corner by complaints and corrections and unable to provide what I wanted, it seemed to get an increased number of broken responses (an error message partway through) or what appeared to be resets (like maybe th…
Those symptoms seem to be common. Another one I have had often is this "reset" you mentioned. One of the most frustrating things is it forgetting our conversation up to that point.
Re: GPT-3.5 crashes when it thinks about useRalativeImagePath too much
#158Earlier quoted context omitted.
1. Your original wording, "getting a response _for_ n tokens", does not parse as "getting a response containing n tokens" to me. 2. Clearly, _you_ don't know the API, as you can get output up to the total context length of any of the GPT-4 32k models. I've received output up to 16k tokens from gpt-4-32k-0613. 3. I am currently violating my own principle of avoiding correcting stupid people on the Internet, which is a…
You might want to see a specialist about your behavioral issues. Also gpt-4-32k is not open to public.
Re: GPT-3.5 crashes when it thinks about useRalativeImagePath too much
#159Earlier quoted context omitted.
1. Your original wording, "getting a response _for_ n tokens", does not parse as "getting a response containing n tokens" to me. 2. Clearly, _you_ don't know the API, as you can get output up to the total context length of any of the GPT-4 32k models. I've received output up to 16k tokens from gpt-4-32k-0613. 3. I am currently violating my own principle of avoiding correcting stupid people on the Internet, which is a…
You might want to see a specialist about your behavioral issues. Also gpt-4-32k is not open to public.
Re: GPT-3.5 crashes when it thinks about useRalativeImagePath too much
#160Earlier quoted context omitted.
How many terabytes of information is that roughly? I wonder what LLMs would look like if they weren't able to be trained on the collective community efforts of Reddit + StackOverflow exports
About 12 TB uncompressed json until the middle of 2022, with a dataset that grows 250GB+ per month. If you throw away all metadata you are left with between half and a quarter of that in high quality text.
That's a hot take