Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: LLMs can generate valid JSON 100% of the time

github.com

Post reply on HN
Print view View original

111–120 of 315 posts

Re: Show HN: LLMs can generate valid JSON 100% of the time

#111

Post by joshuanapoli »

simonw wrote: post #103

I really hope OpenAI add something like this to their endpoints soon. Being able to pass up some kind of grammar (a regular expression, or a JSON schema, or some other format) and have this trick run during their token sampling process to ensure the output was compliant would be incredibly useful.

Isn't the Function Calling feature meant for this purpose? It guides the LLM to output according to the given schema. The name of the feature is a little misleading.

https://platform.openai.com/docs/guides/gpt/function-calling

Re: Show HN: LLMs can generate valid JSON 100% of the time

#112

Post by [deleted] »

lefttoreader wrote: post #110

The “trick” seems to blatantly rip off FlashText without citing it? https://arxiv.org/pdf/1711.00046.pdf I’m a fan of the approach. I normally wouldn’t care if this was just another LLM library taking inspiration, but if you’re going to go out of your way to put a paper on the ArXiv, feels like doing a literature review is a good step?

[deleted]

Re: Show HN: LLMs can generate valid JSON 100% of the time

#113

Post by btwillard »

burke wrote: post #105
Animats wrote: post #25

OK, you get syntactically valid JSON, but does it contain the correct info? This is effectively a polisher, like spell check, which gives the output superficially correct form but doesn't understand the content. Right?

You can go pretty deep once you get context free grammars. For example, I'm using torch-grammar (but outlines should be able to do the same thing once CFG support is merged) to not just restrict the format of a generation to a DSL's syntax, but to restrict the keys it updates to valid keys in a known set. e.g.: int_key ::= DQUO ("f" ("e" ("atured-" ("b" ("log." ("p" ("ost_limit" | "a" ... Obviously, yeah, it doesn't…

FYI: We've had grammar constraints available in Outlines for a while, but not using the FSM and indexing approach that makes the regex case so fast. My open PR only adds that.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#114

Post by [deleted] »

remilouf wrote: post #17
J_Shelby_J wrote: post #15

So to explain this another way: After each token generated by the LLM you update the logit bias “mask” to only allow the next token to be a valid json token? Very slick!

Indeed. And we're able to update the mask with a dictionary lookup instead of looping over the entire vocabulary (slow!).

[deleted]

Re: Show HN: LLMs can generate valid JSON 100% of the time

#115

Post by potatoman22 »

simonw wrote: post #103

I really hope OpenAI add something like this to their endpoints soon. Being able to pass up some kind of grammar (a regular expression, or a JSON schema, or some other format) and have this trick run during their token sampling process to ensure the output was compliant would be incredibly useful.

They recently added logit biases, so that's a start.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#116

Post by tornato7 »

joshuanapoli wrote: post #111
simonw wrote: post #103

I really hope OpenAI add something like this to their endpoints soon. Being able to pass up some kind of grammar (a regular expression, or a JSON schema, or some other format) and have this trick run during their token sampling process to ensure the output was compliant would be incredibly useful.

Isn't the Function Calling feature meant for this purpose? It guides the LLM to output according to the given schema. The name of the feature is a little misleading. https://platform.openai.com/docs/guides/gpt/function-calling

Function Calling is fine-tuned to a certain output format, but it very often strays from that format. My function-calling-handling code has a mess of edge case handlers that catch when GPT-4 is calling functions incorrectly.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#117

Post by Lerc »

chipsrafferty wrote: post #81
coder543 wrote: post #44

Earlier quoted context omitted.

This analogy falls apart because the spellchecker is separate from the author, and doesn’t know what the author intended. Here, the LLM is still dictating the token probabilities, so the content will be as correct as the LLM can make it, given the constraints. AIUI, the sampler is just choosing tokens on a combination of probability and syntactic correctness, instead of strictly on probability. If the LLM is forced t…

Why isn't it possible to design LLMs that say "I don't know"?

They can say I don't know when they contain the fact that they don't know something. For instance saying "I don't know" could be a response to"What is the meaning of life"

On the other hand if you ask a LLM how to do something about fish maintenance that it does not know how to do, it might produce an answer like "Sure, first take your fish and " at which point all of the options for the next word are all over the place because there isn't the information available to guide the choice. The sentence started as if it knew the answer because there was no information to say that it didn't. By the time the absence of information has an impact, the LLM is already committed to the sentence where it is confidently giving you an answer.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#118

Post by simonw »

joshuanapoli wrote: post #111
simonw wrote: post #103

I really hope OpenAI add something like this to their endpoints soon. Being able to pass up some kind of grammar (a regular expression, or a JSON schema, or some other format) and have this trick run during their token sampling process to ensure the output was compliant would be incredibly useful.

Isn't the Function Calling feature meant for this purpose? It guides the LLM to output according to the given schema. The name of the feature is a little misleading. https://platform.openai.com/docs/guides/gpt/function-calling

Surprisingly the function calling mechanism doesn't appear to use this trick - apparently it's still possible to get the wrong JSON structure back from it occasionally.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#119

Post by M4v3R »

joshuanapoli wrote: post #111
simonw wrote: post #103

I really hope OpenAI add something like this to their endpoints soon. Being able to pass up some kind of grammar (a regular expression, or a JSON schema, or some other format) and have this trick run during their token sampling process to ensure the output was compliant would be incredibly useful.

Isn't the Function Calling feature meant for this purpose? It guides the LLM to output according to the given schema. The name of the feature is a little misleading. https://platform.openai.com/docs/guides/gpt/function-calling

It’s not though, they even say it in their docs that sending a schema does not guarantee that the model will actually adhere to the scheme or even produce valid JSON
Post reply on HN