Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Native JSON Output from GPT-4

yonom.substack.com

Post reply on HN
Print view View original

51–60 of 258 posts

Re: Native JSON Output from GPT-4

#51

Post by newhouseb »

sanxiyn wrote: post #14
tornato7 wrote: post #7

Earlier quoted context omitted.

IIRC, there's a way to "force" LLMs to output proper JSON by adding some logic to the top token selection. I.e. in the randomness function (which OpenAI calls temperature) you'd never choose a next token that results in broken JSON. The only reason it wouldn't would be if the output exceeds the token limit. I wonder if OpenAI is doing something like this.

Note that this (token selection restriction) is even available on OpenAI API as logit_bias.

But only for the whole generation. So if you want to constrain things one token at a time (as you would to force things to follow a grammar) you have to make fresh calls and only request one token which makes things more or less impractical if you want true guarantees. A few months ago I built this anyway to suss out how much more expensive it was [1]

[1] https://github.com/newhouseb/clownfish#so-how-do-i-use-this-...

Re: Native JSON Output from GPT-4

#52

Post by smallerfish »

I will experiment with this at the weekend. Once thing I found useful with supplying a json schema in the prompt was that I could supply inline comments and tell it when to leave a field null, etc. I found that much more reliable than describing these nuances elsewhere in the prompt. Presumably I can't do this with functions, but maybe I'll be able to work around it in the prompt (particularly now that I have more room to play with.)

Re: Native JSON Output from GPT-4

#53

Post by lbeurerkellner »

behnamoh wrote: post #44
minimaxir wrote: post #3

After reading the docs for the new ChatGPT function calling yesterday, it's structured and/or typed data for GPT input or output that's the key feature of these new models. The ReAct flow of tool selection that it provides is secondary. As this post notes, you don't even need to the full flow of passing a function result back to the model: getting structured data from ChatGPT in itself has a lot of fun and practical…

What's the implication of this new change for Microsoft Guidance, LMQL, Langchain, etc.? It looks like much of their functionality (controlling model output) just became obsolete. Am I missing something?

If anything this removes a major roadblock for libraries/languages that want to employ LLM calls as a primitive, no? Although, I fear the vendor lock-in intensifies here, also given how restrictive and specific the Chat API.

Either way, as part of the LMQL team, I am actually pretty excited about this, also with respect to what we want to build going forward. This makes language model programming much easier.

Re: Native JSON Output from GPT-4

#54

Post by IshKebab »

irthomasthomas wrote: post #50

It's a shame they couldn't use yaml, instead. I compared them and yaml uses about 20% fewer tokens. However, I can understand accuracy, derived from frequency, being more important than token budget.

I would imagine JSON is easier for a LLM to understand (and for humans!) because it doesn't rely on indentation and confusing syntax for lists, strings etc.

Re: Native JSON Output from GPT-4

#55

Post by yonom »

EGreg wrote: post #47

Actually I'm looking to take GPT-4 output and create file formats like keynote presentations, or pptx. Is that currently possible with some tools?

I would recommend creating a simplified JSON schema for the slides (say, presentation is an array of slides, each slide has a title, body, optional image, optional diagram, each diagram is one of pie, table, ... Then use a library to generate the pptx file from the content generated.

Re: Native JSON Output from GPT-4

#57

Post by lbeurerkellner »

swyx wrote: post #19

i think people are underestimating the potential here for agents building - it is now a lot easier for GPT4 to call other models, or itself. while i was taking notes for our emergency pod yesterday ( https://www.latent.space/p/function-agents ) we had this interesting debate with Simon Willison on just how many functions will be supplied to this API. Simon thinks it will be "deep" rather than "wide" - eg a few functi…

It's interesting to think about this form of computation (LLM + function call) in terms of circuitry. It is still unclear to me however, if the sequential form of reasoning imposed by a sequence of chat messages is the right model here. LLM decoding and also more high-level "reasoning algorithms" like tree of thought are not that linear.

Ever since we started working on LMQL, the overarching vision all along was to get to a form of language model programming, where LLM calls are just the smallest primitive of the "text computer" you are running on. It will be interesting to see what kind of patterns emerge, now that the smallest primitive becomes more robust and reliable, at least in terms of the interface.

Re: Native JSON Output from GPT-4

#58

Post by AdrienBrault »

irthomasthomas wrote: post #50

It's a shame they couldn't use yaml, instead. I compared them and yaml uses about 20% fewer tokens. However, I can understand accuracy, derived from frequency, being more important than token budget.

I think YAML actually uses more tokens than JSON without indents, especially with deep data. For example "," being a single token makes JSON quite compact.

You can compare JSON and YAML on https://platform.openai.com/tokenizer

Re: Native JSON Output from GPT-4

#59

Post by ftxbro »

swyx wrote: post #19

i think people are underestimating the potential here for agents building - it is now a lot easier for GPT4 to call other models, or itself. while i was taking notes for our emergency pod yesterday ( https://www.latent.space/p/function-agents ) we had this interesting debate with Simon Willison on just how many functions will be supplied to this API. Simon thinks it will be "deep" rather than "wide" - eg a few functi…

> "you can now trivially make GPT4 decide whether to call itself again, or to proceed to the next stage."

Does this mean the GPT-4 API is now publicly available, or is there still a waitlist? If there's a waitlist and you literally are not allowed to use it no matter how much you are willing to pay then it seems like it's hard to call that trivial.

Re: Native JSON Output from GPT-4

#60

Post by yonom »

darepublic wrote: post #24

I have been using gpt4 to translate natural language to JSON already. And on v4 ( not v3) it hasn't returned any malformed JSON iirc

- if the only reason you're using v4 over v3.5 is to generate JSON, you can now use this API and downgrade for faster and cheaper API calls. - malicious user input may break your json (by asking GPT to include comments around the JSON, as another user suggested); this may or may not be an issue (e. g. if one user can influence other users' experience)
Post reply on HN