Thanks for building this. The mechanics are such an obvious idea that it's astounding that the first-party platforms haven't done this yet. I would be interested to see how this could be used for other tasks outside of JSON that require structured input.
Show HN: LLMs can generate valid JSON 100% of the time
11–20 of 315 posts
Re: Show HN: LLMs can generate valid JSON 100% of the time
#12Relevant; LLama.cpp implemented grammar-based sampling last month. https://news.ycombinator.com/item?id=36819906 https://github.com/ggerganov/llama.cpp/pull/1773
We can extend our approach to grammar-based sampling, as explained in the paper linked above. Relevant PR: https://github.com/normal-computing/outlines/pull/178
Our method is much more efficient. llama.cpp loops over the entire vocabulary (~50k tokens) at each step to generate the mask. We generate an index at initialization, and building the masks at each step only requires a dictionary lookup (trade speed for memory). Sampling is just as fast as standard sampling.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#13I can make GPT4 return valid JSON simply by providing examples in the system message. This works nine times out of ten. But it's still probabilistic, and nine times out of ten isn't good enough. Occasionally it will hallucinate responses like this: {"key1": "value1", "key2": "value2" for i in range(n)} Re-prompting with the parsing error message is usually enough to get it on the second try. But escaping double-quote…
It took some iterations but I've managed to get the OpenAI API to give me valid JSON 100% of the time now(based on my testing). I think I put in the prompt to never use newlines because it was causing issues lol.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#14Thanks for building this. The mechanics are such an obvious idea that it's astounding that the first-party platforms haven't done this yet. I would be interested to see how this could be used for other tasks outside of JSON that require structured input.
Thanks! We have extended the approach to grammar-based sampling. We describe the approach in the paper linked above. The following PR is relevant: https://github.com/normal-computing/outlines/pull/178
Re: Show HN: LLMs can generate valid JSON 100% of the time
#15So to explain this another way:
After each token generated by the LLM you update the logit bias “mask” to only allow the next token to be a valid json token?
Very slick!
Re: Show HN: LLMs can generate valid JSON 100% of the time
#16Earlier quoted context omitted.
Thanks for bringing this library to my attention! From my understanding, TypeChat proceeds by (1) generating (2) attempting validation (3) if it fails, call the LLM again to fix the output (4) etc. Our method on the other guarantees that the output will follow the specs of the JSON schema. No need to call the LLM several times.
There's also https://lmql.ai/
LQML (and guidance https://github.com/guidance-ai/guidance) are much more inefficient. They loop over the entire vocabulary at each step, we only do it once at initialization.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#17So to explain this another way: After each token generated by the LLM you update the logit bias “mask” to only allow the next token to be a valid json token? Very slick!
Indeed. And we're able to update the mask with a dictionary lookup instead of looping over the entire vocabulary (slow!).
Re: Show HN: LLMs can generate valid JSON 100% of the time
#18This is awesome. I have a vision to build self-managed software. This will be a great tool.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#19This is awesome. I have a vision to build self-managed software. This will be a great tool.
Thank you! Hope this helps and opens many applications :)
Re: Show HN: LLMs can generate valid JSON 100% of the time
#20This is awesome. I have a vision to build self-managed software. This will be a great tool.
This is really great too, I am building self-generating experiments and molecular simulations with https://atomictessellator.com and I am going to try out this framework after work