Recent and related:
Function calling and other API updates - https://news.ycombinator.com/item?id=36313348 - June 2023 (154 comments)
Native JSON Output from GPT-4
31–40 of 258 posts
Re: Native JSON Output from GPT-4
#32Recent and related: Function calling and other API updates - https://news.ycombinator.com/item?id=36313348 - June 2023 (154 comments)
IMO this isn't a dupe and shouldn't be penalized as a result.
Re: Native JSON Output from GPT-4
#33Earlier quoted context omitted.
IIRC, there's a way to "force" LLMs to output proper JSON by adding some logic to the top token selection. I.e. in the randomness function (which OpenAI calls temperature) you'd never choose a next token that results in broken JSON. The only reason it wouldn't would be if the output exceeds the token limit. I wonder if OpenAI is doing something like this.
It would seem not, as the official documentation mentions the arguments may be hallucinated or be a malformed JSON . (except if the meaning is the JSON syntax is valid but may not conform to the schema, but they're unclear on that).
For various reasons, token selection may be implemented as upweighting/downweighting instead of outright ban of invalid tokens. (Maybe it helps training?) Then the model could generate malformed JSON. I think it is premature to infer from "can generate malformed JSON" that OpenAI is not using token selection restriction.
Re: Native JSON Output from GPT-4
#34Re: Native JSON Output from GPT-4
#35Earlier quoted context omitted.
IIRC, there's a way to "force" LLMs to output proper JSON by adding some logic to the top token selection. I.e. in the randomness function (which OpenAI calls temperature) you'd never choose a next token that results in broken JSON. The only reason it wouldn't would be if the output exceeds the token limit. I wonder if OpenAI is doing something like this.
Note that you don’t necessarily need to have the AI output any JSON at all — simply have it answer when being asked for the value to a specific JSON key, and handle the JSON structure part in your hallucinations-free own code: https://github.com/manuelkiessling/php-ai-tool-bridge
Thanks for sharing!
Re: Native JSON Output from GPT-4
#36Building agents that use advanced API's was not really practical until now. Things like Langchain's Structured Agents worked somewhat reliably, but due to the massive token count it was so slow, the experience was _never_ going to be useful.
Due to this, the performance in which our agent processes results has improved 5-6 times and it does actually do a pretty good job of keeping the schema.
One problem that is not resolved yet is that it still hallucinates a lot of attributes. For example we have tool that allows it to create contacts in user's CRM. I ask it to:
"Create contacts for top 3 Barcelona players:.
It creates an structure like this"
1. Lionel Messi - Email: lionel.messi@barcelona.com - Phone Number: +1234567890 - Tags: Player, Barcelona
2. Gerard Pique - Email: gerard.pique@barcelona.com - Phone Number: +1234567891 - Tags: Player, Barcelona
3. Marc-Andre ter Stegen - Email: marc-terstegen@barcelona.com - Phone Number: +1234567892 - Tags: Player, Barcelona
And you can see it hallucinated email addresses and phone numbers.
Re: Native JSON Output from GPT-4
#37Re: Native JSON Output from GPT-4
#38Can I use this to make it reliably output code (say JavaScript)? I haven't managed to do it with just prompt engineering as it will still add explanations, apologies and do other unwanted things like splitting the code into two files as markdown.
Not this, but using the token selection restriction approach, you can let LLM produce output that conforms to arbitrary formal grammar completely reliably. JavaScript, Python, whatever.
Re: Native JSON Output from GPT-4
#39I'm wondering if introducing a system message like "convert the resulting json to yaml and return the yaml only" would adversely affect the optimization done for these models. The reason is that yaml uses significantly fewer tokens compared to json. For the output, where data type specification or adding comments may not be necessary, this could be beneficial. From my understanding, specifying functions in json now uses fewer tokens, but I believe the response still consumes the usual amount of tokens.