Native JSON Output from GPT-4
yonom.substack.com
Native JSON Output from GPT-4
1–10 of 258 posts
Re: Native JSON Output from GPT-4
#2The JSON schema not counting toward token usage is huge, that will really help reduce costs.
Re: Native JSON Output from GPT-4
#3After reading the docs for the new ChatGPT function calling yesterday, it's structured and/or typed data for GPT input or output that's the key feature of these new models. The ReAct flow of tool selection that it provides is secondary.
As this post notes, you don't even need to the full flow of passing a function result back to the model: getting structured data from ChatGPT in itself has a lot of fun and practical use cases. You could coax previous versions of ChatGPT to "output results as JSON" with a system prompt but in practice results are mixed, although even with this finetuned model the docs warn that there still could be parsing errors.
OpenAI's demo for function calling is not a Hello World, to put it mildly: https://github.com/openai/openai-cookbook/blob/main/examples...
Re: Native JSON Output from GPT-4
#4The JSON schema not counting toward token usage is huge, that will really help reduce costs.
I believe functions do count in some way toward the token usage; but it seems to be in a more efficient way than pasting raw JSON schemas into the prompt. Nevertheless, the token usage seems to be far lower than previous alternatives, which is awesome!
Re: Native JSON Output from GPT-4
#5The JSON schema not counting toward token usage is huge, that will really help reduce costs.
That is up in the air and needs more testing. Field descriptions, for example, are important but extraneous input that would be tokenized and count in the costs.
At the least for ChatGPT, input token costs were cut by 25% so it evens out.
Re: Native JSON Output from GPT-4
#6In the openai blog post they mention "Convert “Who are my top ten customers this month?” to an internal API call" but I'm assuming they mean gpt will respond with structured json (we define via schema in function prompt) that we can use to more easily programatically make that api call?
I could be confused but I'm interpreting this function calling as "a way to define structured input and selection of function and then structured output" but not the actual ability to send it arbitrary code to execute.
Still amazing, just wanting to see if I'm wrong on this.
Re: Native JSON Output from GPT-4
#7After reading the docs for the new ChatGPT function calling yesterday, it's structured and/or typed data for GPT input or output that's the key feature of these new models. The ReAct flow of tool selection that it provides is secondary. As this post notes, you don't even need to the full flow of passing a function result back to the model: getting structured data from ChatGPT in itself has a lot of fun and practical…
IIRC, there's a way to "force" LLMs to output proper JSON by adding some logic to the top token selection. I.e. in the randomness function (which OpenAI calls temperature) you'd never choose a next token that results in broken JSON. The only reason it wouldn't would be if the output exceeds the token limit. I wonder if OpenAI is doing something like this.
Re: Native JSON Output from GPT-4
#8Glad we didn't get to far into adopting something like Guardrails. This sort of kills it's main value prop for OpenAI.
Re: Native JSON Output from GPT-4
#9Can I use this to make it reliably output code (say JavaScript)? I haven't managed to do it with just prompt engineering as it will still add explanations, apologies and do other unwanted things like splitting the code into two files as markdown.
Re: Native JSON Output from GPT-4
#10Can I use this to make it reliably output code (say JavaScript)? I haven't managed to do it with just prompt engineering as it will still add explanations, apologies and do other unwanted things like splitting the code into two files as markdown.
Here’s an approach to return just JavaScript:
https://github.com/williamcotton/transynthetical-engine
The key is the addition of few-shot exemplars.