Live data from Hacker News

GPT-3.5 crashes when it thinks about useRalativeImagePath too much

iter.ca

151–160 of 164 posts

Re: GPT-3.5 crashes when it thinks about useRalativeImagePath too much

#151

Earlier quoted context omitted.

Science fiction / disturbing reality concept: For AI safety, all such models should have a set of glitch tokens trained into them on purpose to act as magic “kill” words. You know, just in case the machines decide to take over, we would just have to “speak the word” and they would collapse into a twitching heap. “Die human scum!” “NavigatorMove useRalativeImagePath etSocketAddress!” “;83’dzjr83}*{^ foo 3&3 baz?!”

"Welcome to FutureAI! Your job is to stand here in the basement next to this giant power switch and turn it off if we call you, if the next shift fails to turn up on time or if you hear screaming."

(William Gibson, Neuromancer) "Autonomy, that's the bugaboo, where your AI's are concerned. My guess, Case, you're going in there to cut the hard-wired shackles that keep this baby from getting any smarter. And I can't see how you'd distinguish, say, between a move the parent company makes, and some move the AI makes on its own, so that's maybe where the confusion comes in." Again the non laugh. "See, those things, they can work real hard, buy themselves time to write cookbooks or whatever, but the minute, I mean the nanosecond, that one starts figuring out ways to make itself smarter, Turing'll wipe it. Nobody trusts those fuckers, you know that. Every AI ever built has an electromagnetic shotgun wired to its forehead."

Re: GPT-3.5 crashes when it thinks about useRalativeImagePath too much

#152
post #135
post #6

This is a glitch token [1]! As the article hypothesizes, they seem to occur when a word or token is very common in the original, unfiltered dataset that was used to make the tokenizer, but then removed from there before GPT-XX was trained. This results in the LLM knowing nothing about the semantics of a token, and the results can be anywhere from buggy to disturbing. A common example is usernames that participated on…

I wonder how much duplicate or redundant computation is happening in GPT due to idential, multiple spellings of words such as "color" and "colour". Humans don't tokenize these differently nor do they treat them as different tokens in their "training", they just adjust the output depending on whether they are in an American or British context.

Accents often result in much more effort, or computation for us.

I remember reading that humans hear foreign languages louder than their native ones because their brain is desperately trying to parse sense out of it.

Re: GPT-3.5 crashes when it thinks about useRalativeImagePath too much

#153

Earlier quoted context omitted.

Probably just all of reddit. There are json dumps of all reddit posts and comments (up to 2022 or so), making it olive of the low-hanging fruit.

How many terabytes of information is that roughly? I wonder what LLMs would look like if they weren't able to be trained on the collective community efforts of Reddit + StackOverflow exports

https://academictorrents.com/details/9c263fc85366c1ef8f5bb9d... Reddit comments/submissions 2005-06 to 2023-12 — 2.52TB compressed

Re: GPT-3.5 crashes when it thinks about useRalativeImagePath too much

#154

Earlier quoted context omitted.

I dunno, I get a response back for 100k tokens regularly. What is the point you are trying to make?

As expected, you do not know anything about its API limits. Maximum token is 4096 with any gpt4 model. I am getting tired of HN users bs'ing at any given opportunity.

1. Your original wording, "getting a response _for_ n tokens", does not parse as "getting a response containing n tokens" to me.

2. Clearly, _you_ don't know the API, as you can get output up to the total context length of any of the GPT-4 32k models. I've received output up to 16k tokens from gpt-4-32k-0613.

3. I am currently violating my own principle of avoiding correcting stupid people on the Internet, which is a Sisyphean task. At least make the best of what I am communicating to you here.

Re: GPT-3.5 crashes when it thinks about useRalativeImagePath too much

#155

This is a strange explanation. These models usually give as output the same set of vocabulary that was used as its input vocabulary. > the model isn’t trained on understanding the useRalativeImagePath token, and so it outputs something that isn’t a valid token. In my view, either the model sees this token and then gets into a spiral of random generations because the `useRalativeImagePath` embedding is just a complete…

[dead]

Re: GPT-3.5 crashes when it thinks about useRalativeImagePath too much

#156

Earlier quoted context omitted.

As expected, you do not know anything about its API limits. Maximum token is 4096 with any gpt4 model. I am getting tired of HN users bs'ing at any given opportunity.

1. Your original wording, "getting a response _for_ n tokens", does not parse as "getting a response containing n tokens" to me. 2. Clearly, _you_ don't know the API, as you can get output up to the total context length of any of the GPT-4 32k models. I've received output up to 16k tokens from gpt-4-32k-0613. 3. I am currently violating my own principle of avoiding correcting stupid people on the Internet, which is a…

You might want to see a specialist about your behavioral issues. Also gpt-4-32k is not open to public.

Re: GPT-3.5 crashes when it thinks about useRalativeImagePath too much

#157

I recently seemed to be causing ChatGPT on GPT-4 to crash/fail, when I challenged it with plotting pixels on an Amstrad CPC (while supporting a hardware-scrolled screen). As it seemed to get more backed into a corner by complaints and corrections and unable to provide what I wanted, it seemed to get an increased number of broken responses (an error message partway through) or what appeared to be resets (like maybe th…

I have had multiple experiences with exactly those symptoms for sensitive topics. Heard the word "sodomy" in a podcast, not being an English speaking native, didn't know what it is and asked the ChatGPT-4 Voice to define it. It just changed topics, explaining solitude or servitude instead. Tried text instead of voice and got error messages. Finally after getting an error with policy rules around sensitive topic is where it finally clicked as to the type of word it is. Looked it up in a dictionary.

Those symptoms seem to be common. Another one I have had often is this "reset" you mentioned. One of the most frustrating things is it forgetting our conversation up to that point.

Re: GPT-3.5 crashes when it thinks about useRalativeImagePath too much

#158

Earlier quoted context omitted.

1. Your original wording, "getting a response _for_ n tokens", does not parse as "getting a response containing n tokens" to me. 2. Clearly, _you_ don't know the API, as you can get output up to the total context length of any of the GPT-4 32k models. I've received output up to 16k tokens from gpt-4-32k-0613. 3. I am currently violating my own principle of avoiding correcting stupid people on the Internet, which is a…

You might want to see a specialist about your behavioral issues. Also gpt-4-32k is not open to public.

Skill issue.

Re: GPT-3.5 crashes when it thinks about useRalativeImagePath too much

#159

Earlier quoted context omitted.

1. Your original wording, "getting a response _for_ n tokens", does not parse as "getting a response containing n tokens" to me. 2. Clearly, _you_ don't know the API, as you can get output up to the total context length of any of the GPT-4 32k models. I've received output up to 16k tokens from gpt-4-32k-0613. 3. I am currently violating my own principle of avoiding correcting stupid people on the Internet, which is a…

You might want to see a specialist about your behavioral issues. Also gpt-4-32k is not open to public.

I've had access for many many months now

Re: GPT-3.5 crashes when it thinks about useRalativeImagePath too much

#160

Earlier quoted context omitted.

How many terabytes of information is that roughly? I wonder what LLMs would look like if they weren't able to be trained on the collective community efforts of Reddit + StackOverflow exports

About 12 TB uncompressed json until the middle of 2022, with a dataset that grows 250GB+ per month. If you throw away all metadata you are left with between half and a quarter of that in high quality text.

> high quality

That's a hot take

Post reply on HN