Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Native JSON Output from GPT-4

yonom.substack.com

Post reply on HN
Print view View original

231–240 of 258 posts

Re: Native JSON Output from GPT-4

#231

Post by golol »

andai wrote: post #224
golol wrote: post #117

Earlier quoted context omitted.

Honestly, I suspect asking GPT-4 to fix your JSON (in a new chat) is a good drunken JSON parser. We are only scraping the surface of what's possible with LLMs. If Token generation was free and instant we could come up with a giant schema of interacting model calls that generates 10 suggestions, iterates over them, ranks them and picks the best one, as silly as it sounds.

That's hilarious... if parsing GPT's JSON fails, keep asking GPT to fix it until it parses!

It shouldn't be surprising though. If a human makes an error parsing JSON, what do you do? You make them look over it again. Unless their intelligence is the bottleneck they might just be able to fix it.

Re: Native JSON Output from GPT-4

#232

Post by golergka »

jacobsimon wrote: post #192
sheepscreek wrote: post #164

Earlier quoted context omitted.

The solution that worked great for me - do not use JSON for GPT to agent communication. Use comma separated key=value, or something to that effect. Then have another pure code layer to parse that into structured JSON. I think it’s the JSON syntax (with curly braces) that does it in. So YAML or TOML might work just as well, but I haven’t tried that.

Coincidentally, I just published this JS library[1] over the weekend that helps prompt LLMs to return typed JSON data and validates it for you. Would love feedback on it if this is something people here are interested in. Haven’t played around with the new API yet but I think this is super exciting stuff! [1] https://github.com/jacobsimon/prompting

Looks promising! Do you do retries when returned json is invalid? Personally, I used io-ts for parsing, and GPT seems to be able to correct itself easily when confronted with a well-formed error message.

Re: Native JSON Output from GPT-4

#233

Post by golergka »

SamPatt wrote: post #102
jonplackett wrote: post #85

Earlier quoted context omitted.

Not that it matters now but just doing something like this works 99% of the time or more with 4 and 90% with 3.5. It is VERY IMPORTANT that you respond in valid JSON ONLY. Nothing before or after. Make sure to escape all strings. Use this format: {“some_variable”: [describe the variable purpose]}

99% of the time is still super frustrating when it fails, if you're using it in a consumer facing app. You have to clean up the output to avoid getting an error. If it goes from 99% to 100% JSON that is a big deal for me, much simpler.

If you're building an app based on LLMs that expects higher than 99% correctness from it, you are bound to fail. Negative scenarios workarounds and retries are mandatory.

Re: Native JSON Output from GPT-4

#234

Post by golergka »

andai wrote: post #224
golol wrote: post #117

Earlier quoted context omitted.

Honestly, I suspect asking GPT-4 to fix your JSON (in a new chat) is a good drunken JSON parser. We are only scraping the surface of what's possible with LLMs. If Token generation was free and instant we could come up with a giant schema of interacting model calls that generates 10 suggestions, iterates over them, ranks them and picks the best one, as silly as it sounds.

That's hilarious... if parsing GPT's JSON fails, keep asking GPT to fix it until it parses!

It works. Just be sure to build a good error message.

Re: Native JSON Output from GPT-4

#235

Post by golergka »

nasir wrote: post #67
irthomasthomas wrote: post #50

It's a shame they couldn't use yaml, instead. I compared them and yaml uses about 20% fewer tokens. However, I can understand accuracy, derived from frequency, being more important than token budget.

Its a lot more straightforward to use JSON programmatically than YAML.

If you are using any kind of type checking instead of blindly trusting generated json it's exactly the same amount of work.

Re: Native JSON Output from GPT-4

#236

Post by edelans »

edwin wrote: post #41

For those who want to test out the LLM as API idea, we are building a turnkey prompt to API product. Here's Simon's recipe maker deployed in a minute: https://preview.promptjoy.com/apis/1AgCy9 . Public preview to make and test your own API: https://preview.promptjoy.com

Congrats on the first-time user experience, I could experiment with your API in a few seconds, and the product is sleek!

Re: Native JSON Output from GPT-4

#237

Post by sheepscreek »

bombela wrote: post #191
sheepscreek wrote: post #164

Earlier quoted context omitted.

The solution that worked great for me - do not use JSON for GPT to agent communication. Use comma separated key=value, or something to that effect. Then have another pure code layer to parse that into structured JSON. I think it’s the JSON syntax (with curly braces) that does it in. So YAML or TOML might work just as well, but I haven’t tried that.

It's harder to form a tree with key value. I also tried the relational route. But it would always messup the cardinality (one person should have 0 or n friends, but a person has a single birth date).

You could flatten it using namespaced keys. Eg.

    {
      parent1: { child1: value }
    }
Becomes one of the following:

    parent1/child1=value
    parent1_child1=value
    parent1.child1=value
..you get the idea.

Re: Native JSON Output from GPT-4

#238

Post by jacobsimon »

golergka wrote: post #232
jacobsimon wrote: post #192

Earlier quoted context omitted.

Coincidentally, I just published this JS library[1] over the weekend that helps prompt LLMs to return typed JSON data and validates it for you. Would love feedback on it if this is something people here are interested in. Haven’t played around with the new API yet but I think this is super exciting stuff! [1] https://github.com/jacobsimon/prompting

Looks promising! Do you do retries when returned json is invalid? Personally, I used io-ts for parsing, and GPT seems to be able to correct itself easily when confronted with a well-formed error message.

Great idea, I was going to add basic retries but didn’t think to include the error.

Any other features you’d expect in a prompt builder like this? I’m tempted to add lots of other utility methods like classify(), summarize(), language(), etc

Re: Native JSON Output from GPT-4

#239

Post by gawi »

neuronexmachina wrote: post #222
gawi wrote: post #173

Earlier quoted context omitted.

It's only been added to the OpenAI interface. Function calling is really useful when used with agents. To include that to agents would require some redesign as the tool instructions should be removed from the prompt templates in favor of function definitions in the API request. The response parsing code would also be affected. I just hope they won't come up with yet another agent type.

Like this? https://github.com/hwchase17/langchain/blob/master/langchain...

LangChain is a perpetual hackathon.

Re: Native JSON Output from GPT-4

#240

Post by yunyu »

ttul wrote: post #187
tornato7 wrote: post #7

Earlier quoted context omitted.

IIRC, there's a way to "force" LLMs to output proper JSON by adding some logic to the top token selection. I.e. in the randomness function (which OpenAI calls temperature) you'd never choose a next token that results in broken JSON. The only reason it wouldn't would be if the output exceeds the token limit. I wonder if OpenAI is doing something like this.

I think the problem is that tokens are not characters. So even if you had access to a JSON parser state that could tell you whether or not a given character is valid as the next character, I am not sure how you would translate that into tokens to apply the logit biases appropriately. There would be a great deal of computation required at each step to scan the parser state and generate the list of prohibited or allowa…

You simply sample tokens starting with the allowed characters and truncate if needed. It’s pretty efficient, there’s an implementation here: https://github.com/1rgs/jsonformer
Post reply on HN