Viewing profile — goodside

More or less; they’re not literally the same tokens as “a”, “b”, “c” but I’d speculate the mapping is learned from some other examples of ASCII (or just Roman letters) being repeat…

(I’m the person interviewed in the article.) The trick is Unicode code points are only assigned individual tokens if they’re nontrivially used outside of some other already tokeniz…

Machine translation is valuable and it’s all neural these days.

Narragansett Hydraulic Rower. The oars use pressurized gas to provide resistance.

Thanks, this is great — excerpted here: https://twitter.com/goodside/status/1747088701694370274

No, in both tokenizers Unicode tag-block code points like these are converted into bytes (two tokens per character), which is a fallback for code points uncommon enough to not warr…

For anyone unable to view X/Twitter posts: Unfortunately Hacker News comments strip emojis, and in this example the suffix instructions are hiding as tag-block characters after an …

It’s not just a matter of the tokenization being the same, it’s whether the model can understand text that’s written with a very rarely seen encoding. Normally tokens represent ent…

Unfortunately Hacker News comments strip emojis, and in this example the hidden text is hiding as tag-block characters after an emoji (which are typically part of the emoji, used f…

The only intended difference I’m aware of is that answers on mobile are more concise, but the varying system instructions will affect demos like this one.

I can’t imagine it was intentionally added as a feature. It doesn’t work in GPT-3.5 — it seems GPT-4 is unexpectedly smart enough to parse the invisible portion (and confuse it for…

In my tests GPT-3.5 just isn’t smart enough to parse the hidden text encoding. It’s encoded in a way that’s programmatically trivial to convert to ASCII but text written this way u…

How this works: A suffix of text is written using non-printing Unicode tag block characters that correspond to ASCII. GPT-4 is able understand these because the mapping to ASCII is…

It’s non-printing Unicode tag block characters that correspond directly to ASCII and the AI is able understand them. It’s a deprecated feature of Unicode once used to encode the la…

The API isn’t ChatGPT. The underlying model is the same but ChatGPT uses system instructions that vary by platform, and supports browsing, DALL-E, and code execution. It’s most rel…

The OpenAI Playground isn’t ChatGPT, it’s the more raw API. The underlying model is the same but ChatGPT uses system instructions that vary by platform, and supports browsing, DALL…

The web UI (which I forgot to test) appears to be affected by different implicit system instructions, which allow e.g. more concise answers on mobile. Try the mobile app. Other exa…

Add to your custom instructions: "DO NOT BROWSE THE WEB unless explicitly requested."

I’d quibble with #1 slightly — prompt injection is an attack whoever otherwise controls the model, regardless of whether that party a human. We think of SQL injection as an attack …

This post is AI-generated spam. There’s whole paragraphs that lose track of the fact this is about LLM prompting and talk about prompting humans with body language.

That's true, but it was expensive and until recently you could only tune older versions of GPT-3 lacking both instruction tuning and the code pre-training of the Codex models (from…

You do, because it’s not just more training it’s PPO updates instead of MLE. It’s no longer trying to estimate the token distribution of the training corpus, it’s trying to shift l…