Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Native JSON Output from GPT-4

yonom.substack.com

Post reply on HN
Print view View original

71–80 of 258 posts

Re: Native JSON Output from GPT-4

#71

Post by edwin »

yonom wrote: post #63
edwin wrote: post #41

For those who want to test out the LLM as API idea, we are building a turnkey prompt to API product. Here's Simon's recipe maker deployed in a minute: https://preview.promptjoy.com/apis/1AgCy9 . Public preview to make and test your own API: https://preview.promptjoy.com

This is cool! Are you using one-shot learning under the hood with a user provided example?

BTW: Here's a more performant version (fewer tokens) https://preview.promptjoy.com/apis/jNqCA2 that uses a smaller example but will still generate pretty good results.

Re: Native JSON Output from GPT-4

#72

Post by cjonas »

minimaxir wrote: post #25
swyx wrote: post #19

i think people are underestimating the potential here for agents building - it is now a lot easier for GPT4 to call other models, or itself. while i was taking notes for our emergency pod yesterday ( https://www.latent.space/p/function-agents ) we had this interesting debate with Simon Willison on just how many functions will be supplied to this API. Simon thinks it will be "deep" rather than "wide" - eg a few functi…

"Trivial" is misleading. From OpenAI's docs and demos, the full ReAct workflow is an order of magnitude more difficult than typical ChatGPT API usage with a new set of constaints (e.g. schema definitions) Even OpenAI's notebook demo has error handling workflows which was actually necessary since ChatGPT returned incorrect formatted output.

Maybe trivial isn't the right word, but it's still very straight-forward to get something basic, yet really powerful...

ReAct Setup Prompt (goal + available actions) -> Agent "ReAction" -> Parse & Execute Action -> Send Action Response (success or error) -> Agent "ReAction" -> repeat

As long as each action has proper validation and returns meaningful error messages, you don't need to even change the control flow. The agent will typically understand what went wrong, and attempt to correct it in the next "ReAction".

I've been refactoring some agents to use "functions" and so far it seems to be a HUGE improvement in reliability vs the "Return JSON matching this format" approach. Most impactful is that fact that "3.5-turbo" will now reliability return JSON (before you'd be forced to use GPT-4 for an ReAct style agent of modest complexity).

My agents also seem to be better at following other instructions now that the noise of the response format is gone (of course it's still there, but in a way it has been specifically trained on). This could also just be a result of the improvements to the system prompt though.

Re: Native JSON Output from GPT-4

#73

Post by ilaksh »

aecorredor wrote: post #69

Newbie in machine learning here. It’s crazy that this is the top post just today. I’ve been doing the intro to deep learning course from MIT this week, mainly because I have a ton of JSON files that are already classified, and want to train a model that can generate new JSON data by taking classification tags as input. So naturally this post is exciting. My main unknown right now is figuring out which model to train…

Did you read the article? To do it with OpenAI you would just put a few output examples in the prompt and then give it a function that takes the class and the output parameters correspond to the JSON format you want, or just a string containing JSON.

You could also fine tuned an LLM like Falcon-7b but probably not necessary and nothing to do with OpenAI.

You might also look into the OpenAI Embedding API as a third option.

I would try the first option though.

Re: Native JSON Output from GPT-4

#74

Post by cwxm »

jonplackett wrote: post #65
swyx wrote: post #19

i think people are underestimating the potential here for agents building - it is now a lot easier for GPT4 to call other models, or itself. while i was taking notes for our emergency pod yesterday ( https://www.latent.space/p/function-agents ) we had this interesting debate with Simon Willison on just how many functions will be supplied to this API. Simon thinks it will be "deep" rather than "wide" - eg a few functi…

It was already quite easy to get GPT-4 to output json. You just append ‘reply in json with this format’ and it does a really good job. GPT-3.5 was very haphazard though and needs extensive babysitting and reminding, so if this makes gpt3 better then it’s useful - it does have an annoying disclaimer though that ‘it may not reply with valid json’ so we’ll still have to do some sense checks into he output. I have been u…

even with gpt 4, it hallucinates enough that it’s not reliable, forgetting to open/close brackets and quotes. This sounds like it’d be a big improvement.

Re: Native JSON Output from GPT-4

#75

Post by edwin »

adultSwim wrote: post #27

Running an LLM every time someone clicks on a button is expensive and slow in production, but probably still ~10x cheaper to produce than code.

New techniques like semantic caching will help. This is the modern era's version of building a performant social graph.

Re: Native JSON Output from GPT-4

#76

Post by m3kw9 »

It works pretty good. You define a few “function” and enter a description on what it does, when user prompts, it will understand the prompt and tell you which likely “function” to use, which is just the function name. I feel like this is a new way to program, a sort of fuzzy logic type of programming

Re: Native JSON Output from GPT-4

#77

Post by ztratar »

cwxm wrote: post #74
jonplackett wrote: post #65

Earlier quoted context omitted.

It was already quite easy to get GPT-4 to output json. You just append ‘reply in json with this format’ and it does a really good job. GPT-3.5 was very haphazard though and needs extensive babysitting and reminding, so if this makes gpt3 better then it’s useful - it does have an annoying disclaimer though that ‘it may not reply with valid json’ so we’ll still have to do some sense checks into he output. I have been u…

even with gpt 4, it hallucinates enough that it’s not reliable, forgetting to open/close brackets and quotes. This sounds like it’d be a big improvement.

Nah, this was solved by most teams a while ago.

Re: Native JSON Output from GPT-4

#78

Post by iamflimflam1 »

It’s pretty interesting how the work they’ve been doing on plugins has fed into this.

I suspect that they’ve managed to get a lot of good training data by calling the APIs provided by plugins and detecting when it’s gone wrong from bad request responses.

Re: Native JSON Output from GPT-4

#79

Post by daralthus »

edwin wrote: post #75
adultSwim wrote: post #27

Running an LLM every time someone clicks on a button is expensive and slow in production, but probably still ~10x cheaper to produce than code.

New techniques like semantic caching will help. This is the modern era's version of building a performant social graph.

What's semantic caching?

Re: Native JSON Output from GPT-4

#80

Post by ilaksh »

swyx wrote: post #19

i think people are underestimating the potential here for agents building - it is now a lot easier for GPT4 to call other models, or itself. while i was taking notes for our emergency pod yesterday ( https://www.latent.space/p/function-agents ) we had this interesting debate with Simon Willison on just how many functions will be supplied to this API. Simon thinks it will be "deep" rather than "wide" - eg a few functi…

The thing is the relevant context often depends on what it's trying to do. You can give it a lot of context in 16k but if there are too many different types of things then I think it will be confused or at least have less capacity for the actual selected task.

So what I am thinking is that some functions might just be like gateways into a second menu level. So instead of just edit_file with the filename and new source, maybe only select_files_for_edit is available at the top level. In that case I can ensure it doesn't try to overwrite an existing file without important stuff that was already in there, by providing the requested files existing contents along with the function allowing the file edit.

Post reply on HN