Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Native JSON Output from GPT-4

yonom.substack.com

Post reply on HN
Print view View original

211–220 of 258 posts

Re: Native JSON Output from GPT-4

#211

Post by gregw134 »

037 wrote: post #39

I'm wondering if introducing a system message like "convert the resulting json to yaml and return the yaml only" would adversely affect the optimization done for these models. The reason is that yaml uses significantly fewer tokens compared to json. For the output, where data type specification or adding comments may not be necessary, this could be beneficial. From my understanding, specifying functions in json now u…

That's what I'm doing. I ask ChatGPT to return inline yaml (no wasting tokens on line breaks), then I parse the yaml output into JSON once I receive it. A bit awkward but it cuts costs in half.

Re: Native JSON Output from GPT-4

#212

Post by quickthrower2 »

swyx wrote: post #19

i think people are underestimating the potential here for agents building - it is now a lot easier for GPT4 to call other models, or itself. while i was taking notes for our emergency pod yesterday ( https://www.latent.space/p/function-agents ) we had this interesting debate with Simon Willison on just how many functions will be supplied to this API. Simon thinks it will be "deep" rather than "wide" - eg a few functi…

The first transistors were slow, and it seems this "GPT3/4 calling itself" stuff is quite slow. GPT3/4 as a direct chat is about as slow as I can take. Once this gets sped up.

I am sure it will, as you can scale out, scale up and build more efficient code and build more efficient architectures and "tool for the job" different parts of the process.

The problem now (using auto gpt, for example) is accuracy is bad, so you need human feedback and intervention AND it is slow. Take away the slow, or the needing human intervention and this can be very powerful.

I dream of the breakthrough "shitty old laptop is all you need" paper where they figure out how to do amazing stuff with a 1Gb of space on a spinny disk and 1Gb RAM and a CPU.

Re: Native JSON Output from GPT-4

#213

Post by LanceJones »

whateveracct wrote: post #169
bel423 wrote: post #145

Earlier quoted context omitted.

It literally does it everytime perfectly. I remember I put together an entire system that would validate the JSON against a zod schema and use reflection to fix it and it literally never gets triggered because GPT3.5-turbo always does it right the first time.

No it doesn't lol. I've seen it just randomly not use a comma after one array element, for example.

Yep. Incorrect trailing commas ad nauseum for me.

Re: Native JSON Output from GPT-4

#214

Post by boringuser2 »

delhanty wrote: post #200
boringuser2 wrote: post #175

Earlier quoted context omitted.

Do you people always have to overhype this shit?

Do you have to be nasty? That's a person you're replying to with feelings, so why not default to being kind in comments as per HN guidelines? As it happens, swyx has built notable AI related things, for example smol-developer https://twitter.com/swyx/status/1657892220492738560 and it would be nice to be able to read his and other perspectives without having to read shallow, mean, dismissive replies such as yours.

[flagged]

Re: Native JSON Output from GPT-4

#215

Post by danShumway »

BoorishBears wrote: post #209
danShumway wrote: post #207

Earlier quoted context omitted.

> e.g. define a function called extract_data(name: string, birthday: string), or sql_query(query: string) This section in OpenAI's product announcement really irritates me because it's so obvious that the model should have access to a subset of API calls that themselves fetch the data, as opposed to giving the model raw access to SQL. You could have the same capabilities while eliminating a huge amount of risk. And O…

You don't need to directly run the query it returns, you can use that query as a sub-query on a known safe set of data and let it fail if someone manages to prompt inject their way into looking at other tables/columns. That way you can support natural language to query without sending dozens of functions (which will eat up the context window)

You can do that (I wouldn't advise it, there are still problems that are better solved by building explicit functions; but you can use subqueries and it would be safer) -- but most developers won't. They'll run the query directly. Most developers also will not execute it as a readonly query, they'll give the LLM write access to the database.

If OpenAI doesn't know that, then I don't know what to say, they haven't spent enough time writing documentation for general users.

Re: Native JSON Output from GPT-4

#216

Post by kristiandupont »

abhibeckert wrote: post #204
danShumway wrote: post #199

I'm concerned that OpenAI's example documentation suggests using this to A) construct SQL queries and B) summarize emails, but that their example code doesn't include clear hooks for human validation before actions are called. For a recipe builder it's not so big a deal, but I really worry how eager people are to remove human review from these steps. It gets rid of a very important mechanism for reducing the risks of…

In my opinion the only way to use it safely is to ensure your AI only has access to data that the end user already has access to. At that point, prompt injection is no-longer an issue - because the AI doesn't need to hide anything. Giving GPT access to your entire database, but telling it not to reveal certain bits, is never going to work. There will always be side channel vulnerabilities in those systems.

>At that point, prompt injection is no-longer an issue [...]

As far as input goes, yes. But I am more worried about agents that can take actions that affect the outside world, like sending emails on your behalf.

Re: Native JSON Output from GPT-4

#217

Post by kristiandupont »

sillysaurusx wrote: post #205
danShumway wrote: post #199

I'm concerned that OpenAI's example documentation suggests using this to A) construct SQL queries and B) summarize emails, but that their example code doesn't include clear hooks for human validation before actions are called. For a recipe builder it's not so big a deal, but I really worry how eager people are to remove human review from these steps. It gets rid of a very important mechanism for reducing the risks of…

I was going to say “I look forward to it and think it’s hilarious,” but then I remembered that most victims will be people learning to code, not companies. It would really suck to suddenly lose your recipe database when you just wanted to figure out how this programming stuff worked. Some kind of “heads up” tagline is probably a good idea, yeah.

I think the victims will mostly be the users of the software. The personal assistant that can handle your calendar and emails and all would be able to do real damage.

Re: Native JSON Output from GPT-4

#218

Post by SimianLogic »

swyx wrote: post #19

i think people are underestimating the potential here for agents building - it is now a lot easier for GPT4 to call other models, or itself. while i was taking notes for our emergency pod yesterday ( https://www.latent.space/p/function-agents ) we had this interesting debate with Simon Willison on just how many functions will be supplied to this API. Simon thinks it will be "deep" rather than "wide" - eg a few functi…

I agree with this. We’ve already gotten pretty good at json coercion, but this seems like it goes one step further by bundling decision making in to the model instead of junking up your prompt or requiring some kind of eval on a single json response.

It should also be much easier to cache these functions. If you send the same set of functions on every API hit, OpenAI should be able to cache that more intelligently than if everything was one big text prompt.

Re: Native JSON Output from GPT-4

#219

Post by jimmySixDOF »

ehsanu1 wrote: post #155
fumar wrote: post #134

Earlier quoted context omitted.

It is “good enough”. Where I struggle is maintaining its memory through a longer request where multiple iterations fail or succeed and then all of a sudden its memory is exceeded and starts fresh. I wish I could store “learnings” that it could revisit.

Sounds like you want something like tree of thoughts: https://arxiv.org/abs/2305.10601

Interestingly the paper's repo starts off :

Blah Blah "...is NOT the correct implementation to replicate paper results. In fact, people have reported that his code cannot properly run, and is probably automatically generated by ChatGPT, and kyegomez has done so for other popular ML methods, while intentionally refusing to link to official implementations for his own interests"

Love a good GitHub Identity Theft Star farming ML story

But this method could have potential for a chain of function

Re: Native JSON Output from GPT-4

#220

Post by jillesvangurp »

swyx wrote: post #19

i think people are underestimating the potential here for agents building - it is now a lot easier for GPT4 to call other models, or itself. while i was taking notes for our emergency pod yesterday ( https://www.latent.space/p/function-agents ) we had this interesting debate with Simon Willison on just how many functions will be supplied to this API. Simon thinks it will be "deep" rather than "wide" - eg a few functi…

Exactly, we humans can use specialized models and traditional tool APIs and models and orchestrate the use of all these without understanding how these things work in detail.

To do accounting, GPT 4 (or future models) doesn't have to know how to calculate. All it needs to know how to interface with tools like calculators, spreadsheets, etc. and parse their outputs. Every script, program, etc. becomes a thing that has such an API. A lot what we humans do to solve problems is breaking down big problems into problems where we know the solution already.

Real life tool interfaces are messy and optimized for humans with their limited language and cognitive skills. Ironically, that means they are relatively easy to figure out for AI language models. Relative to human language the grammar of these tool "languages" is more regular and the syntax less ambiguous and complicated. Which is why gpt 3 and 4 are reasonably proficient with even some more obscure programming languages and in the use of various frameworks; including some very obscure ones.

Given a lot of these tools with machine accessible APIs with some sort of description or documentation, figuring out how to call these things is relatively straightforward for a language model. The rest is just coming up with a high level plan and then executing it. Which amounts to generating some sort of script that does this. As soon as you have that, that in itself becomes a tool that may be used later. So, it can get better over time. Especially once it starts incorporating feedback about the quality of its results. It would be able to run mini experiments and run its own QA on its own output as well.

Post reply on HN