Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Native JSON Output from GPT-4

yonom.substack.com

Post reply on HN
Print view View original

191–200 of 258 posts

Re: Native JSON Output from GPT-4

#191

Post by bombela »

sheepscreek wrote: post #164
jonplackett wrote: post #65

Earlier quoted context omitted.

It was already quite easy to get GPT-4 to output json. You just append ‘reply in json with this format’ and it does a really good job. GPT-3.5 was very haphazard though and needs extensive babysitting and reminding, so if this makes gpt3 better then it’s useful - it does have an annoying disclaimer though that ‘it may not reply with valid json’ so we’ll still have to do some sense checks into he output. I have been u…

The solution that worked great for me - do not use JSON for GPT to agent communication. Use comma separated key=value, or something to that effect. Then have another pure code layer to parse that into structured JSON. I think it’s the JSON syntax (with curly braces) that does it in. So YAML or TOML might work just as well, but I haven’t tried that.

It's harder to form a tree with key value. I also tried the relational route. But it would always messup the cardinality (one person should have 0 or n friends, but a person has a single birth date).

Re: Native JSON Output from GPT-4

#192

Post by jacobsimon »

sheepscreek wrote: post #164
jonplackett wrote: post #65

Earlier quoted context omitted.

It was already quite easy to get GPT-4 to output json. You just append ‘reply in json with this format’ and it does a really good job. GPT-3.5 was very haphazard though and needs extensive babysitting and reminding, so if this makes gpt3 better then it’s useful - it does have an annoying disclaimer though that ‘it may not reply with valid json’ so we’ll still have to do some sense checks into he output. I have been u…

The solution that worked great for me - do not use JSON for GPT to agent communication. Use comma separated key=value, or something to that effect. Then have another pure code layer to parse that into structured JSON. I think it’s the JSON syntax (with curly braces) that does it in. So YAML or TOML might work just as well, but I haven’t tried that.

Coincidentally, I just published this JS library[1] over the weekend that helps prompt LLMs to return typed JSON data and validates it for you. Would love feedback on it if this is something people here are interested in. Haven’t played around with the new API yet but I think this is super exciting stuff!

[1] https://github.com/jacobsimon/prompting

Re: Native JSON Output from GPT-4

#193

Post by lmeyerov »

bel423 wrote: post #145
ignite wrote: post #106

Earlier quoted context omitted.

> You just append ‘reply in json with this format’ and it does a really good job. It does an ok job. Except when it doesn't. Definitely misses a lot of the time, sometimes on prompts that succeeded on previous runs.

It literally does it everytime perfectly. I remember I put together an entire system that would validate the JSON against a zod schema and use reflection to fix it and it literally never gets triggered because GPT3.5-turbo always does it right the first time.

Yeah no

Re: Native JSON Output from GPT-4

#194

Post by pyinstallwoes »

boringuser2 wrote: post #190
throwuwu wrote: post #181

Earlier quoted context omitted.

What’s your problem? There’s nothing overhyped about that comment. People, including me, are building complex agents that can execute multi stage prompts and perform complex tasks. Comparing these first models to a basic unit of logic is more than fair given how much more capable they are. Do you just have an axe to grind?

[flagged]

How is it inappropriate? How is it not building?

Re: Native JSON Output from GPT-4

#196

Post by khazhoux »

I'm trying to experiment with the API but the response time is always in the 15-25second range. How are people getting any interesting work done with it?

I see others on the OpenAPI dev forum complaining about this too, but no resolution.

Re: Native JSON Output from GPT-4

#197

Post by wonderfuly »

edwin wrote: post #41

For those who want to test out the LLM as API idea, we are building a turnkey prompt to API product. Here's Simon's recipe maker deployed in a minute: https://preview.promptjoy.com/apis/1AgCy9 . Public preview to make and test your own API: https://preview.promptjoy.com

I own this domain: prompts.run Do you wanna it?

Re: Native JSON Output from GPT-4

#198

Post by lyjackal »

ManuelKiessling wrote: post #20
tornato7 wrote: post #7

Earlier quoted context omitted.

IIRC, there's a way to "force" LLMs to output proper JSON by adding some logic to the top token selection. I.e. in the randomness function (which OpenAI calls temperature) you'd never choose a next token that results in broken JSON. The only reason it wouldn't would be if the output exceeds the token limit. I wonder if OpenAI is doing something like this.

Note that you don’t necessarily need to have the AI output any JSON at all — simply have it answer when being asked for the value to a specific JSON key, and handle the JSON structure part in your hallucinations-free own code: https://github.com/manuelkiessling/php-ai-tool-bridge

Would be nice if you could send a back and forth interaction for each key. This approach turns into lots of requests that reapply the entire context and ends up slow. I wish i could just send a Microsoft guidance template program, and process that in a single pass.

Re: Native JSON Output from GPT-4

#199

Post by danShumway »

I'm concerned that OpenAI's example documentation suggests using this to A) construct SQL queries and B) summarize emails, but that their example code doesn't include clear hooks for human validation before actions are called.

For a recipe builder it's not so big a deal, but I really worry how eager people are to remove human review from these steps. It gets rid of a very important mechanism for reducing the risks of prompt injection.

The top comment here suggests wiring this up to allow GPT-4 to recursively call itself. Meanwhile, some of the best advice I've seen from security professionals on secure LLM app development is to whenever possible completely isolate queries from each other to reduce the potential damage that a compromised agent can do before its "memory" is wiped.

There are definitely ways to use this safely, and there are definitely some pretty powerful apps you could build on top of this without much risk. LLMs as a transformation layer for trusted input is a good use-case. But are devs going to stick with that? Is it going to be used safely? Do devs understand any of the risks or how to mitigate them in the first place?

3rd-party plugins on ChatGPT have repeatedly been vulnerable in the real world, I'm worried about what mistakes developers are going to make now that they're actively encouraged to treat GPT as even more of a low-level data layer. Especially since OpenAI's documentation on how to build secure apps is mostly pretty bad, and they don't seem to be spending much time or effort educating developers/partners on how to approach LLM security.

Re: Native JSON Output from GPT-4

#200

Post by delhanty »

boringuser2 wrote: post #175
swyx wrote: post #19

i think people are underestimating the potential here for agents building - it is now a lot easier for GPT4 to call other models, or itself. while i was taking notes for our emergency pod yesterday ( https://www.latent.space/p/function-agents ) we had this interesting debate with Simon Willison on just how many functions will be supplied to this API. Simon thinks it will be "deep" rather than "wide" - eg a few functi…

Do you people always have to overhype this shit?

Do you have to be nasty?

That's a person you're replying to with feelings, so why not default to being kind in comments as per HN guidelines?

As it happens, swyx has built notable AI related things, for example smol-developer

https://twitter.com/swyx/status/1657892220492738560

and it would be nice to be able to read his and other perspectives without having to read shallow, mean, dismissive replies such as yours.

Post reply on HN