Does this work in tandem with beam search or does it do greedy sampling?
Show HN: LLMs can generate valid JSON 100% of the time
41–50 of 315 posts
Re: Show HN: LLMs can generate valid JSON 100% of the time
#42Thanks for building this. The mechanics are such an obvious idea that it's astounding that the first-party platforms haven't done this yet. I would be interested to see how this could be used for other tasks outside of JSON that require structured input.
> it's astounding that the first-party platforms haven't done this yet
I was under the impression LLM tech is currently in a breakneck arms race and that things are dramatically changing every few months. It could simply just be a consequence of limited developer resources. It would be "astounding" if decade-old tech were missing such a fundamental feature, but for AI tech in arms-race mode it seems reasonable that they are still missing QoL features.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#43Does this work in tandem with beam search or does it do greedy sampling?
The underlying approach can improve the performance of anything that requires the set of non-zero probability tokens at each step, and anything that needs to continue matching/parsing from a previous state.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#44OK, you get syntactically valid JSON, but does it contain the correct info? This is effectively a polisher, like spell check, which gives the output superficially correct form but doesn't understand the content. Right?
This analogy falls apart because the spellchecker is separate from the author, and doesn’t know what the author intended.
Here, the LLM is still dictating the token probabilities, so the content will be as correct as the LLM can make it, given the constraints. AIUI, the sampler is just choosing tokens on a combination of probability and syntactic correctness, instead of strictly on probability.
If the LLM is forced to provide a numeric temperature for Seattle, and the input doesn’t contain that data, then obviously the LLM will be forced by the sampler to provide a random answer if the sampler will accept nothing else, much like a human who is forced to mark “true”/“false” on an online form, with no option to reject the question and explain that the question isn’t even a true/false question.
I don’t know about this specific implementation, but it seems important to design systems like this to always “accept” (sample for) an error response from the LLM so that it can hopefully reject invalid requests.
But, yes, all the usual caveats about LLMs apply. It can’t provide correct answers to things it doesn’t know. Forcing it to respond with the answer to the life, the universe, and everything is not going to provide a meaningful response. Even things it “knows”, it can still get wrong sometimes.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#45Earlier quoted context omitted.
> maybe running a schema validator against the output and re-requesting on your behalf this is the part that blows my mind. You don't have to do this! You don't have to sample the entire output, and then validate after the fact. You're not required to greedily pick the token with the highest score. You get the scores of all tokens, on every forward pass! So why even waste time picking invalid tokens if you're just go…
Maybe re-read what I said? > realistically it's probably cheaper to just re-emit than to add the machinery that enables this to their existing architecture There are literally dozens of random projects that have implemented logit based masking, it's a trivial thing to implement. What's probably not as trivial is deploying it at scale with whatever architecture OpenAI already has in place. Especially if they're using…
when GPT-4 first became available, I had a feeling that something about it felt “hacky”. Compared to GPT-3 which was more streamlined, mature, and well thought out, GPT-4 was like a system put together to outperform the previous one at all costs. I wouldn’t be surprised if that led to design decisions that made their model hard to improve. Maybe GPT-5 will not be around any time soon.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#46"Generating valid JSON" is not impressive. Here's some valid JSON: []
The tricky part is generating useful JSON.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#47So to explain this another way: After each token generated by the LLM you update the logit bias “mask” to only allow the next token to be a valid json token? Very slick!
It’s actually a very old trick. Lots of libraries do this. idk what’s the big deal about this one.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#48Earlier quoted context omitted.
Good catch! It really is a combination of guidance guaranteeing JSON output and OpenAI getting it right a good majority of the time[0]. But yeah, I can see how it can be frustrating that the JSON output is not guaranteed by the docs. [0] >>99% in my experience
That said, I am definitely going to look into this library and compare its results to guidance, since they claim it blows it out of the water (which is very enticing!)
Figure 2 in our paper (https://arxiv.org/abs/2307.09702) shows the difference for a single regex.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#49How does this compare in terms of latency, cost, and effectiveness to jsonformer? https://github.com/1rgs/jsonformer
Re: Show HN: LLMs can generate valid JSON 100% of the time
#50So to explain this another way: After each token generated by the LLM you update the logit bias “mask” to only allow the next token to be a valid json token? Very slick!
It’s actually a very old trick. Lots of libraries do this. idk what’s the big deal about this one.
Perhaps I didn’t explain clearly enough in the original post?