Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: LLMs can generate valid JSON 100% of the time

github.com

Post reply on HN
Print view View original

91–100 of 315 posts

Re: Show HN: LLMs can generate valid JSON 100% of the time

#91

Post by phillipcarter »

panarky wrote: post #8

I can make GPT4 return valid JSON simply by providing examples in the system message. This works nine times out of ten. But it's still probabilistic, and nine times out of ten isn't good enough. Occasionally it will hallucinate responses like this: {"key1": "value1", "key2": "value2" for i in range(n)} Re-prompting with the parsing error message is usually enough to get it on the second try. But escaping double-quote…

This is what we do, but for GPT-3.5. And it doesn't need to be system messages either. We even have it emitting only JSON in a specific structure (except for when it fails to produce an output altogether). This is without the function calling model.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#92

Post by Lerc »

remilouf wrote: post #14
xigency wrote: post #11

Thanks for building this. The mechanics are such an obvious idea that it's astounding that the first-party platforms haven't done this yet. I would be interested to see how this could be used for other tasks outside of JSON that require structured input.

Thanks! We have extended the approach to grammar-based sampling. We describe the approach in the paper linked above. The following PR is relevant: https://github.com/normal-computing/outlines/pull/178

Could this same approach be applied at training? If the guidance does a lot of the syntactical heavy lifting, would that create the opportunity for a model to use the weights for something else. Essentially not bothering to reduce the error of things that the guidance will stomp on anyway.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#93

Post by superasn »

orasis wrote: post #88
panarky wrote: post #8

I can make GPT4 return valid JSON simply by providing examples in the system message. This works nine times out of ten. But it's still probabilistic, and nine times out of ten isn't good enough. Occasionally it will hallucinate responses like this: {"key1": "value1", "key2": "value2" for i in range(n)} Re-prompting with the parsing error message is usually enough to get it on the second try. But escaping double-quote…

What about using ChatGPT’s new function calling mechanism?

That returns broken JSON a lot of the times too

Re: Show HN: LLMs can generate valid JSON 100% of the time

#94

Post by coder543 »

anticrymactic wrote: post #90
coder543 wrote: post #44

Earlier quoted context omitted.

This analogy falls apart because the spellchecker is separate from the author, and doesn’t know what the author intended. Here, the LLM is still dictating the token probabilities, so the content will be as correct as the LLM can make it, given the constraints. AIUI, the sampler is just choosing tokens on a combination of probability and syntactic correctness, instead of strictly on probability. If the LLM is forced t…

I'm stupid with LLMs, but would it be possible to have this output with gpt4's intelligence, or would it have to be specifically trained?

It’s something OpenAI should really implement themselves. Implementing it from the client side will mean sending the same request over and over until you get a syntactically correct answer, which is going to be much slower and likely to cost a lot. The server can guide the generation, but the client can (currently) only hint at what it wants. ChatGPT4 is fairly good at following schemas, and that’s what OpenAI currently relies on, but they make no guarantees.

It likely wouldn’t require additional training. It’s a change to the way the server uses the model, not a change to the model itself… but we don’t know ChatGPT4’s true architecture because OpenAI won’t publish anything about it, so it’s hard to say for sure.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#95

Post by nextaccountic »

panarky wrote: post #8

I can make GPT4 return valid JSON simply by providing examples in the system message. This works nine times out of ten. But it's still probabilistic, and nine times out of ten isn't good enough. Occasionally it will hallucinate responses like this: {"key1": "value1", "key2": "value2" for i in range(n)} Re-prompting with the parsing error message is usually enough to get it on the second try. But escaping double-quote…

What about reprompting with a different temperature value?

If this works, how to select the optimal value? Maybe you can train a model that can excel at the task of querying gpt4 for valid jsons

Re: Show HN: LLMs can generate valid JSON 100% of the time

#96

Post by simonw »

panarky wrote: post #8

I can make GPT4 return valid JSON simply by providing examples in the system message. This works nine times out of ten. But it's still probabilistic, and nine times out of ten isn't good enough. Occasionally it will hallucinate responses like this: {"key1": "value1", "key2": "value2" for i in range(n)} Re-prompting with the parsing error message is usually enough to get it on the second try. But escaping double-quote…

That re-prompting on error trick is what this new Microsoft library does, too: https://github.com/microsoft/TypeChat

Here's their prompt for that: https://github.com/microsoft/TypeChat/blob/c45460f4030938da3...

I think the approach using grammars (seen here, but also in things like https://github.com/ggerganov/llama.cpp/pull/1773 ) is a much more elegant solution.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#97

Post by padjo »

MuffinFlavored wrote: post #70
panarky wrote: post #8

I can make GPT4 return valid JSON simply by providing examples in the system message. This works nine times out of ten. But it's still probabilistic, and nine times out of ten isn't good enough. Occasionally it will hallucinate responses like this: {"key1": "value1", "key2": "value2" for i in range(n)} Re-prompting with the parsing error message is usually enough to get it on the second try. But escaping double-quote…

I wonder if the next iteration of OpenAI features is something like: right now you can inject prompts that the LLM takes into consideration before the output I wonder if you can make it have a "post" generation function that says like "keep re-trying in a loop (aka hallucinating with randomness) until the output message passes XYZ format/checks/scoring"

It’s starting to feel like LLMs are to “classical” software engineering what quantum physics was to classical physics

Re: Show HN: LLMs can generate valid JSON 100% of the time

#98

Post by gowld »

padolsey wrote: post #64
panarky wrote: post #8

I can make GPT4 return valid JSON simply by providing examples in the system message. This works nine times out of ten. But it's still probabilistic, and nine times out of ten isn't good enough. Occasionally it will hallucinate responses like this: {"key1": "value1", "key2": "value2" for i in range(n)} Re-prompting with the parsing error message is usually enough to get it on the second try. But escaping double-quote…

I've had more luck with getting it to output XML as (1) You can imbue XML with actual language/meaning (which LLMs adore) and (2) parsers can be made to be more forgiving. I get why people want to make JSON, but to me it's a bit like trying to get a cat to swim - you might eventually succeed, but it's not their natural inclination.

How do you imbue XML with meaning?

Re: Show HN: LLMs can generate valid JSON 100% of the time

#99

Post by burke »

remilouf wrote: post #12
sneedchucker wrote: post #2

Relevant; LLama.cpp implemented grammar-based sampling last month. https://news.ycombinator.com/item?id=36819906 https://github.com/ggerganov/llama.cpp/pull/1773

We can extend our approach to grammar-based sampling, as explained in the paper linked above. Relevant PR: https://github.com/normal-computing/outlines/pull/178 Our method is much more efficient. llama.cpp loops over the entire vocabulary (~50k tokens) at each step to generate the mask. We generate an index at initialization, and building the masks at each step only requires a dictionary lookup (trade speed for memor…

Yes! This is closer to the approach I took in my port of llama.cpp's grammar support to PyTorch: https://github.com/Shopify/torch-grammar/blob/main/torch_gra... ... it generates a tensor mapping each PDA stack to a map of which tokens are acceptable from that state. It seems like a much better way to do it than looping over the sampled tokens on each turn.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#100

Post by burke »

remilouf wrote: post #28
BoorishBears wrote: post #21

I'm not sure how this is different than: https://github.com/1rgs/jsonformer or https://github.com/newhouseb/clownfish or https://github.com/mkuchnik/relm or https://github.com/ggerganov/llama.cpp/pull/1773 or https://github.com/Shopify/torch-grammar Overall there are a ton of these logit based guidance systems, the reason they don't get tons of traction is the SOTA models are behind REST APIs that don't enable this f…

Thanks for bringing clownfish and relm to my attention! afaik other libraries loop over the entire vocabulary at every step of the generation. We on the other hand build an index at initialization by looping once over the vocabulary. Then generation is just as fast as standard generation.

torch-grammar generates a mask per PDA stack... we don't try to compute all the possible stacks. I'm sure there's something smarter that could be done here and you've probably figured it out (though IIRC regular languages don't have the arbitrarily recursive stack problem that you get when you get to context-free languages?) anyway, in practice we spend a few milliseconds on the first few requests building caches and then just apply masks from caches after that.
Post reply on HN