Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatML: ChatGPT API expects a structured format, called Chat Markup Language

github.com

Post reply on HN
Print view View original

61–70 of 83 posts

Re: ChatML: ChatGPT API expects a structured format, called Chat Markup Language

#61

Post by grncdr »

gdb wrote: post #4

(I work at OpenAI.) This document is a preview of the underlying format consumed by ChatGPT models. As an API user, today you use our higher-level API ( https://platform.openai.com/docs/guides/chat ). We'll be opening up direct access to this format in the future, and want to give people visibility into what's going on under the hood in the meanwhile!

Could you clarify whether the JSON format shown here is really intended to be used by developers vs. the "chat format" shown in https://platform.openai.com/docs/api-reference/chat/create ?

The "chat format" looks simple, extensible, and clean. The JSON format shown in https://github.com/openai/openai-python/blob/main/chatml.md looks ad-hoc, confusing, and (as noted by others) likely to lead to mistakes and injections.

Re: ChatML: ChatGPT API expects a structured format, called Chat Markup Language

#62

Post by blensor »

grncdr wrote: post #61
gdb wrote: post #4

(I work at OpenAI.) This document is a preview of the underlying format consumed by ChatGPT models. As an API user, today you use our higher-level API ( https://platform.openai.com/docs/guides/chat ). We'll be opening up direct access to this format in the future, and want to give people visibility into what's going on under the hood in the meanwhile!

Could you clarify whether the JSON format shown here is really intended to be used by developers vs. the "chat format" shown in https://platform.openai.com/docs/api-reference/chat/create ? The "chat format" looks simple, extensible, and clean. The JSON format shown in https://github.com/openai/openai-python/blob/main/chatml.md looks ad-hoc, confusing, and (as noted by others) likely to lead to mistakes and injections…

I tried it with their python library and that expects a list of dicts with role and content fields. And that seems to translate 1:1 to the API call where it's also expecting that and not chatml markup

Re: ChatML: ChatGPT API expects a structured format, called Chat Markup Language

#63

Post by stefan_ »

Jensson wrote: post #53
friendzis wrote: post #46

Earlier quoted context omitted.

> SQL injection attacks plagued the industry for decades precisely because the initial format didn't think through how to escape queries. No. SQL injection vulnerabilities plagued the industry for decades, as opposed to months/years, because developers thought they can take input in one format, "escape" it enough, sprinkle with addslashes and things will work. And apparently we still teach this even when we have deca…

> because developers thought they can take input in one format, "escape" it enough, sprinkle with addslashes and things will work But that is exactly what the solution is, you escape user strings, there is no other solution to the problem. Either you do it yourself or you use a library to do it, but the end result is the same, I'm not sure why you think this is impossible to do when it has been done successfully for…

No, escaping is precisely not what you do. Escaping is the hack you add because you didn't consider separating code and data in the first place.

Do not offer an API that mixes these two things.

Re: ChatML: ChatGPT API expects a structured format, called Chat Markup Language

#64

Post by shepardrtc »

creatonez wrote: post #19

Why do you think Bing Chat is going in a slightly different direction and not using this format exactly? Their prompt - https://old.reddit.com/r/bing/comments/11bd91j/release_of_th... Wouldn't it be better to unify around ChatML, for the sake of all future training data being consistent? I thought it was strange that they used but not the rest of the ChatML syntax. (There is always the possibility that they are using…

Microsoft has always been very siloed. They want to do their own thing with it - as evidenced by the pile of hot garbage they've turned it into.

Re: ChatML: ChatGPT API expects a structured format, called Chat Markup Language

#66

Post by sc3pt1c »

sillysaurusx wrote: post #34
gdb wrote: post #30

Earlier quoted context omitted.

One detail you may have missed — "system" is only special when it comes right after a special token. So it's not a special token itself, but you cannot inject a valid-looking system message from user text. In more detail, the current format is: HEADER BODY We are actually going to swap over to this shortly: HEADER BODY So basically getting rid of the newline separator and replacing with a special token. Shouldn't cha…

I'm a little confused with your response, or we appear to be talking past each other. For context, I'm a former pentester (NCC Group, formerly Matasano). I've been an ML researcher for four years now, so it's possible I have a unique perspective on this; the combination of pentester + ML is probably rare enough that few others have it. > You cannot inject a valid-looking system message from user text. https://greshak…

This reads like a sales pitch for NCC Group's services.

Re: ChatML: ChatGPT API expects a structured format, called Chat Markup Language

#67

Post by alexwebb2 »

culiao wrote: post #65

What is the best way for a response to come back simply as an integer? So asking, what is the avg. cost of a bagel in new york? I've been able to do it by asking for no text - but wondering if there is an alternative i'm missing.

Few shot learning for sure

“Answer in USD, rounded to the closest integer.

# EXAMPLES What is the average price of a banana? 10

What is the average price of a ticket to see a Star War movie? 1

# QUESTION What is the average price of a bagel in New York?

# ANSWER “

Re: ChatML: ChatGPT API expects a structured format, called Chat Markup Language

#68

Post by alexwebb2 »

gdb wrote: post #8
explaininjs wrote: post #6

Is it just me or is this the least intuitive format imaginable? The type def is something like: type Message = string type Speaker = 'system' | 'user' | 'assistant' | 'system name=example_user' | 'system name=example_assistant' type CML = ('\n' | '${Speaker}\n${Message}' | {token: ' '|' '})[] I'd expect something more like... type Message = string type Speaker = 'system' | 'user' | 'assistant' | 'example_user' | 'exa…

Will all make more sense with upcoming releases, we have a lot of extensions in the works :).

When this is extended to have multiple system roles as designated agents, with mechanisms for the assistant to ping a specific agent for more information or completion of a subtask so devs can route that to secondary AIs or services, that’s going to be a very big deal.

Is that what you’re building toward here?

Re: ChatML: ChatGPT API expects a structured format, called Chat Markup Language

#69

Post by friendzis »

Jensson wrote: post #53
friendzis wrote: post #46

Earlier quoted context omitted.

> SQL injection attacks plagued the industry for decades precisely because the initial format didn't think through how to escape queries. No. SQL injection vulnerabilities plagued the industry for decades, as opposed to months/years, because developers thought they can take input in one format, "escape" it enough, sprinkle with addslashes and things will work. And apparently we still teach this even when we have deca…

> because developers thought they can take input in one format, "escape" it enough, sprinkle with addslashes and things will work But that is exactly what the solution is, you escape user strings, there is no other solution to the problem. Either you do it yourself or you use a library to do it, but the end result is the same, I'm not sure why you think this is impossible to do when it has been done successfully for…

Escaping/sanitizing is required when providing "command+data" inputs to external engines. It's error prone. One needs rigorous escaping done just before the output. Multiple escapes can clash.

> But that is exactly what the solution is, you escape user strings, there is no other solution to the problem

The correct way is to use interfaces that allow separation of command and data inputs. With SQL prepared statements are used. With HTTP data is put in request body or at least after the ?. With HTML data URLs are used. And so on.

> The problem is that many fail to escape strings correctly, not that it is impossible to do.

I really don't want to argue whether escaping correctly is possible at all. Every possible substring sequence, escaping attempts included, that can be interpreted as command by the interpreting system must be accounted for. I would rather avoid the problem altogether, if possible.

Re: ChatML: ChatGPT API expects a structured format, called Chat Markup Language

#70

Post by naushit »

gdb wrote: post #4

(I work at OpenAI.) This document is a preview of the underlying format consumed by ChatGPT models. As an API user, today you use our higher-level API ( https://platform.openai.com/docs/guides/chat ). We'll be opening up direct access to this format in the future, and want to give people visibility into what's going on under the hood in the meanwhile!

Whats wrong with XMPP? Why re-invent the wheel?
Post reply on HN