Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatML: ChatGPT API expects a structured format, called Chat Markup Language

github.com

Post reply on HN
Print view View original

51–60 of 83 posts

Re: ChatML: ChatGPT API expects a structured format, called Chat Markup Language

#51

Post by wongarsu »

pksebben wrote: post #47
joe_the_user wrote: post #32

Earlier quoted context omitted.

You can't (??) make the LLM treat a token deterministically, at least not in my understanding of the current architectures. I believe that's the case and, well, there are some problems there. Specifically, it may be an API but the magic happens with this token response, which is nondeterministic and no controllable, as commentator sillysaurusx notes. IE, you're saying "they're doing anything like security 'cause they…

Not that I can think of an implementation off the top of my head, but there's gotta be non-ai ways to sanitize input before it even hits the model. perhaps I'm just showing my ignorance if the problem space...

You can filter out the string [system], just how in SQL you can escape any quotes. The problem is that it's easy to forget this step somewhere (just as happened with Bing Chat, which filters [system] in chat but not in websites), and you have to cover all possible ways to circumvent your filter. In SQL that was unusual things that also got interpreted as quotes, in LLMs that might be base64-encoding your prompt, and counting on the model to decode it on its own and still recognize the string [system] as special.

Re: ChatML: ChatGPT API expects a structured format, called Chat Markup Language

#52

Post by monkeydust »

mwint wrote: post #12
gdb wrote: post #8

Earlier quoted context omitted.

Will all make more sense with upcoming releases, we have a lot of extensions in the works :).

Somehow, seeing OpenAI employees adding smilies just makes the sense of impending doom even stronger

I am trying to work out the odds of the OpenAI employee not being human.

Re: ChatML: ChatGPT API expects a structured format, called Chat Markup Language

#53

Post by Jensson »

friendzis wrote: post #46
sillysaurusx wrote: post #14

Earlier quoted context omitted.

There doesn't seem to be any way to protect against prompt injection attacks against [system], since [system] isn't a separate token. I understand this is a preview, but if there's one takeaway from the history of cybersecurity attacks, it's this: please put some thought into how queries are escaped. SQL injection attacks plagued the industry for decades precisely because the initial format didn't think through how t…

> SQL injection attacks plagued the industry for decades precisely because the initial format didn't think through how to escape queries. No. SQL injection vulnerabilities plagued the industry for decades, as opposed to months/years, because developers thought they can take input in one format, "escape" it enough, sprinkle with addslashes and things will work. And apparently we still teach this even when we have deca…

> because developers thought they can take input in one format, "escape" it enough, sprinkle with addslashes and things will work

But that is exactly what the solution is, you escape user strings, there is no other solution to the problem. Either you do it yourself or you use a library to do it, but the end result is the same, I'm not sure why you think this is impossible to do when it has been done successfully for decades.

The problem is that many fail to escape strings correctly, not that it is impossible to do.

Re: ChatML: ChatGPT API expects a structured format, called Chat Markup Language

#54

Post by sebzim4500 »

going_ham wrote: post #24
gdb wrote: post #4

(I work at OpenAI.) This document is a preview of the underlying format consumed by ChatGPT models. As an API user, today you use our higher-level API ( https://platform.openai.com/docs/guides/chat ). We'll be opening up direct access to this format in the future, and want to give people visibility into what's going on under the hood in the meanwhile!

Not to sound rude, but how are you guys going to determine differences between user input and say, an input from an external sources like pdf, email, webpage, webapps? Do you have thoughts on it? If I make an application, I will want to link to external systems. If there isn’t any way to distinguish it, I bet the attack surface is too large. If it is restricted to QA without external interface, then usability is also…

From what I can see of the format, there are special tokens (imStart and imEnd) which never appear in external sources.

Re: ChatML: ChatGPT API expects a structured format, called Chat Markup Language

#55

Post by ralusek »

monkeydust wrote: post #52
mwint wrote: post #12

Earlier quoted context omitted.

Somehow, seeing OpenAI employees adding smilies just makes the sense of impending doom even stronger

I am trying to work out the odds of the OpenAI employee not being human.

Just say you think it's human. Saying you think it's not makes them add 50 million parameters, each time.

Re: ChatML: ChatGPT API expects a structured format, called Chat Markup Language

#56

Post by rnosov »

sillysaurusx wrote: post #34
gdb wrote: post #30

Earlier quoted context omitted.

One detail you may have missed — "system" is only special when it comes right after a special token. So it's not a special token itself, but you cannot inject a valid-looking system message from user text. In more detail, the current format is: HEADER BODY We are actually going to swap over to this shortly: HEADER BODY So basically getting rid of the newline separator and replacing with a special token. Shouldn't cha…

I'm a little confused with your response, or we appear to be talking past each other. For context, I'm a former pentester (NCC Group, formerly Matasano). I've been an ML researcher for four years now, so it's possible I have a unique perspective on this; the combination of pentester + ML is probably rare enough that few others have it. > You cannot inject a valid-looking system message from user text. https://greshak…

From reading the docs it looks like there are ( or will be soon ) two distinct ways for API endpoint to consume the prompt:

1. Old one when all inputs are just concatenated into one string (Vulnerable to prompt injection)

2. Inputs supplied separately as a JSON (?) array, so special tokens can be properly encoded, maybe user input stripped of newlines (potentially preventing prompt injection).

I guess when Microsoft were rushing Bing features and faced with a dilemma to do by the rules or by tomorrow they chose the latter.

Re: ChatML: ChatGPT API expects a structured format, called Chat Markup Language

#57

Post by interleave »

staticautomatic wrote: post #17

In what way is this a “list of dicts”? It’s an implicitly structured list of dict, str pairs. And it is WEIRD.

I agree it seems weird on first contact.

To me it makes a bit of sense because I had built a GPT3-based chat two weeks ago, based on their completion documentation^1

The "format" for this prompt already has everything needed: System prompt and history of messages.

```

The following is a conversation with an AI assistant. The assistant is helpful, creative, clever, and very friendly.

Human: Hello, who are you?

AI: I am an AI created by OpenAI. How can I help you today?

Human: Please translate "Hello" to German

AI:

```

To retain context, I kept the system prompt on top and added the prior prompt/completions to the history stack in-between.

End-user-experience-wise, this worked really, really well. Practically as good as the ChatGPT UI.

As a developer though, it was a lot of fiddling, splicing, and dicing of strings. But again, this worked well and I would have kept it as-is for as long as possible.

When I saw the ChatML format, it wasn't a giant leap.

```

system

You are ChatGPT, a large language model trained by OpenAI. Answer as concisely as possible. Knowledge cutoff: 2021-09-01 Current date: 2023-03-01

user

Hello, who are you?

assistant

I am an AI created by OpenAI. How can I help you today?

user

Please translate "Hello" to German

```

I've used that, too. Handling of prior conversations became a bit more manageable. But to honest, there's a lot of markup to keep track of.

To me, the less-discussed JSON format^2 works best. (I'm building a macOS app in Swift)

```

{

  "model": "gpt-3.5-turbo",
  "messages": [

 {
  "role": "system", 
  "content": "You are a helpful and succinct assistant AI created by OpenAI."
 },

 {
  "role": "user", 
  "content": "Hello, who are you?"
 },

 {
  "role": "assistant", 
  "content": "I am an AI created by OpenAI. How can I help you today?"
 },

 {
  "role": "user", 
  "content": "Please translate "Hello" to German"
 }
}

```

What do you think?

^1: https://platform.openai.com/docs/guides/completion/conversat...

^2: https://platform.openai.com/docs/api-reference/chat/create

Re: ChatML: ChatGPT API expects a structured format, called Chat Markup Language

#58

Post by rnosov »

minimaxir wrote: post #36
gdb wrote: post #30

Earlier quoted context omitted.

One detail you may have missed — "system" is only special when it comes right after a special token. So it's not a special token itself, but you cannot inject a valid-looking system message from user text. In more detail, the current format is: HEADER BODY We are actually going to swap over to this shortly: HEADER BODY So basically getting rid of the newline separator and replacing with a special token. Shouldn't cha…

> "system" is only special when it comes right after a special token Can't the user provide the special tokens + system to confuse the model?

If I understood docs correctly to create a special token you'd need to supply it in a special array (which users presumably have no access to).

Re: ChatML: ChatGPT API expects a structured format, called Chat Markup Language

#59

Post by blowski »

Jensson wrote: post #53
friendzis wrote: post #46

Earlier quoted context omitted.

> SQL injection attacks plagued the industry for decades precisely because the initial format didn't think through how to escape queries. No. SQL injection vulnerabilities plagued the industry for decades, as opposed to months/years, because developers thought they can take input in one format, "escape" it enough, sprinkle with addslashes and things will work. And apparently we still teach this even when we have deca…

> because developers thought they can take input in one format, "escape" it enough, sprinkle with addslashes and things will work But that is exactly what the solution is, you escape user strings, there is no other solution to the problem. Either you do it yourself or you use a library to do it, but the end result is the same, I'm not sure why you think this is impossible to do when it has been done successfully for…

What about parameterised queries?

Re: ChatML: ChatGPT API expects a structured format, called Chat Markup Language

#60

Post by sebzim4500 »

gdb wrote: post #4

(I work at OpenAI.) This document is a preview of the underlying format consumed by ChatGPT models. As an API user, today you use our higher-level API ( https://platform.openai.com/docs/guides/chat ). We'll be opening up direct access to this format in the future, and want to give people visibility into what's going on under the hood in the meanwhile!

While you're here, should we expect to be able to finetune gpt-3.5-turbo in the near future? Or are there technical reasons why this is impossible?
Post reply on HN