Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Native JSON Output from GPT-4

yonom.substack.com

Post reply on HN
Print view View original

181–190 of 258 posts

Re: Native JSON Output from GPT-4

#181

Post by throwuwu »

boringuser2 wrote: post #175
swyx wrote: post #19

i think people are underestimating the potential here for agents building - it is now a lot easier for GPT4 to call other models, or itself. while i was taking notes for our emergency pod yesterday ( https://www.latent.space/p/function-agents ) we had this interesting debate with Simon Willison on just how many functions will be supplied to this API. Simon thinks it will be "deep" rather than "wide" - eg a few functi…

Do you people always have to overhype this shit?

What’s your problem? There’s nothing overhyped about that comment. People, including me, are building complex agents that can execute multi stage prompts and perform complex tasks. Comparing these first models to a basic unit of logic is more than fair given how much more capable they are. Do you just have an axe to grind?

Re: Native JSON Output from GPT-4

#182

Post by throwuwu »

jonplackett wrote: post #65
swyx wrote: post #19

i think people are underestimating the potential here for agents building - it is now a lot easier for GPT4 to call other models, or itself. while i was taking notes for our emergency pod yesterday ( https://www.latent.space/p/function-agents ) we had this interesting debate with Simon Willison on just how many functions will be supplied to this API. Simon thinks it will be "deep" rather than "wide" - eg a few functi…

It was already quite easy to get GPT-4 to output json. You just append ‘reply in json with this format’ and it does a really good job. GPT-3.5 was very haphazard though and needs extensive babysitting and reminding, so if this makes gpt3 better then it’s useful - it does have an annoying disclaimer though that ‘it may not reply with valid json’ so we’ll still have to do some sense checks into he output. I have been u…

Just end your request with

‘’’json

Or provide a few examples of user request and then agent response in json. Or both.

Re: Native JSON Output from GPT-4

#183

Post by jarulraj »

swyx wrote: post #19

i think people are underestimating the potential here for agents building - it is now a lot easier for GPT4 to call other models, or itself. while i was taking notes for our emergency pod yesterday ( https://www.latent.space/p/function-agents ) we had this interesting debate with Simon Willison on just how many functions will be supplied to this API. Simon thinks it will be "deep" rather than "wide" - eg a few functi…

Interesting observation, @swyx. There seems to be a connection to transitive closure in SQL queries, where the output of the query is fed as the input to the query in the next iteration [1]. We are thinking about how to best support such recursive functions in EvaDB [2].

[1] http://dwhoman.com/blog/sql-transitive-closure.html [2] https://evadb.readthedocs.io/en/stable/source/tutorials/11-s...

Re: Native JSON Output from GPT-4

#184

Post by devbent »

reallymental wrote: post #91
jonplackett wrote: post #65

Earlier quoted context omitted.

It was already quite easy to get GPT-4 to output json. You just append ‘reply in json with this format’ and it does a really good job. GPT-3.5 was very haphazard though and needs extensive babysitting and reminding, so if this makes gpt3 better then it’s useful - it does have an annoying disclaimer though that ‘it may not reply with valid json’ so we’ll still have to do some sense checks into he output. I have been u…

Is there any publicly available resource replicate your work? I would love to just find the right kind of "incantation" for the gpt-3.5-t or gpt-4 to output a meaningful story arc etc. Any examples of your work would be greatly helpful as well!

I have an open source project doing exactly this at https://www.generativestorytelling.ai/ GitHub link is on the main page!

Re: Native JSON Output from GPT-4

#185

Post by devbent »

cjonas wrote: post #72
minimaxir wrote: post #25

Earlier quoted context omitted.

"Trivial" is misleading. From OpenAI's docs and demos, the full ReAct workflow is an order of magnitude more difficult than typical ChatGPT API usage with a new set of constaints (e.g. schema definitions) Even OpenAI's notebook demo has error handling workflows which was actually necessary since ChatGPT returned incorrect formatted output.

Maybe trivial isn't the right word, but it's still very straight-forward to get something basic, yet really powerful... ReAct Setup Prompt (goal + available actions) -> Agent "ReAction" -> Parse & Execute Action -> Send Action Response (success or error) -> Agent "ReAction" -> repeat As long as each action has proper validation and returns meaningful error messages, you don't need to even change the control flow. The…

For 3.5, I found it easiest to specify a simple, but parsable, format for responses and then convert that to JSON myself.

I'll have to see if the new JSON schema support is easier than what I already have in place.

Re: Native JSON Output from GPT-4

#186

Post by clbrmbr »

throwuwu wrote: post #182
jonplackett wrote: post #65

Earlier quoted context omitted.

It was already quite easy to get GPT-4 to output json. You just append ‘reply in json with this format’ and it does a really good job. GPT-3.5 was very haphazard though and needs extensive babysitting and reminding, so if this makes gpt3 better then it’s useful - it does have an annoying disclaimer though that ‘it may not reply with valid json’ so we’ll still have to do some sense checks into he output. I have been u…

Just end your request with ‘’’json Or provide a few examples of user request and then agent response in json. Or both.

Does the ```json trick work with the chat models? Or only the earlier completion models?

Re: Native JSON Output from GPT-4

#187

Post by ttul »

tornato7 wrote: post #7
minimaxir wrote: post #3

After reading the docs for the new ChatGPT function calling yesterday, it's structured and/or typed data for GPT input or output that's the key feature of these new models. The ReAct flow of tool selection that it provides is secondary. As this post notes, you don't even need to the full flow of passing a function result back to the model: getting structured data from ChatGPT in itself has a lot of fun and practical…

IIRC, there's a way to "force" LLMs to output proper JSON by adding some logic to the top token selection. I.e. in the randomness function (which OpenAI calls temperature) you'd never choose a next token that results in broken JSON. The only reason it wouldn't would be if the output exceeds the token limit. I wonder if OpenAI is doing something like this.

I think the problem is that tokens are not characters. So even if you had access to a JSON parser state that could tell you whether or not a given character is valid as the next character, I am not sure how you would translate that into tokens to apply the logit biases appropriately. There would be a great deal of computation required at each step to scan the parser state and generate the list of prohibited or allowable tokens.

But if one could pull this off, it would be super cool. Similar to how Microsoft’s guidance module uses the logit_bias parameter to force the model to choose between a set of available options.

Re: Native JSON Output from GPT-4

#188

Post by throwuwu »

ilaksh wrote: post #80
swyx wrote: post #19

i think people are underestimating the potential here for agents building - it is now a lot easier for GPT4 to call other models, or itself. while i was taking notes for our emergency pod yesterday ( https://www.latent.space/p/function-agents ) we had this interesting debate with Simon Willison on just how many functions will be supplied to this API. Simon thinks it will be "deep" rather than "wide" - eg a few functi…

The thing is the relevant context often depends on what it's trying to do. You can give it a lot of context in 16k but if there are too many different types of things then I think it will be confused or at least have less capacity for the actual selected task. So what I am thinking is that some functions might just be like gateways into a second menu level. So instead of just edit_file with the filename and new sourc…

Not sure that’s true. I haven’t completely filled the context with examples but I do provide 8 or so exchanges between user and assistant along with a menu of available commands and it seems to be able to generalize from that very well. No hallucinations either. Good idea about sub menus though, I’ll have to use that.

Re: Native JSON Output from GPT-4

#189

Post by throwuwu »

clbrmbr wrote: post #186
throwuwu wrote: post #182

Earlier quoted context omitted.

Just end your request with ‘’’json Or provide a few examples of user request and then agent response in json. Or both.

Does the ```json trick work with the chat models? Or only the earlier completion models?

Works with chat. They’re still text completion models under all that rlhf

Re: Native JSON Output from GPT-4

#190

Post by boringuser2 »

throwuwu wrote: post #181
boringuser2 wrote: post #175

Earlier quoted context omitted.

Do you people always have to overhype this shit?

What’s your problem? There’s nothing overhyped about that comment. People, including me, are building complex agents that can execute multi stage prompts and perform complex tasks. Comparing these first models to a basic unit of logic is more than fair given how much more capable they are. Do you just have an axe to grind?

[flagged]
Post reply on HN