"Generating valid JSON" is not impressive. Here's some valid JSON: [] The tricky part is generating useful JSON.
Or JSON that correctly answers what the prompt is asking.
Show HN: LLMs can generate valid JSON 100% of the time
71–80 of 315 posts
Re: Show HN: LLMs can generate valid JSON 100% of the time
#72Earlier quoted context omitted.
Figure 2 in our paper ( https://arxiv.org/abs/2307.09702 ) shows the difference between guidance and outlines to generate a sequence that is valid to a regex. Jsonformer uses the same technique as guidance. Extrapolate this to several fields. Note that we still need to manage the KV cache in outlines. It’s a small interface change that will be made this week hopefully, but we’ve been focusing on constrained generatio…
Sad to see that my related work on token-level constrained text generation is not cited in the paper: https://github.com/Hellisotherpeople/Constrained-Text-Genera... https://aclanthology.org/2022.cai-1.2/
We're unfortunately only human and didn't catch every single paper on the topic while writing the draft. Thanks for bringing it to our attention.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#73I can make GPT4 return valid JSON simply by providing examples in the system message. This works nine times out of ten. But it's still probabilistic, and nine times out of ten isn't good enough. Occasionally it will hallucinate responses like this: {"key1": "value1", "key2": "value2" for i in range(n)} Re-prompting with the parsing error message is usually enough to get it on the second try. But escaping double-quote…
Yeah same thing. I have done the same with GPT-3.5. Simply ask it to output using provided schema only and give a few examples. Always outputs in provided json format
Re: Show HN: LLMs can generate valid JSON 100% of the time
#74[deleted]
Re: Show HN: LLMs can generate valid JSON 100% of the time
#75So to explain this another way: After each token generated by the LLM you update the logit bias “mask” to only allow the next token to be a valid json token? Very slick!
You would also need to keep generating until the whole string is valid. And what if it gets caught in a loop? Not sure how this can really guarantee 100%
> And what if it gets caught in a loop? Not sure how this can really guarantee 100%
It's not great but after some timeout you can just set the mask to only include closing brackets.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#76OpenAI has this capability built in with functions[0], I believe! Building my own project[1] I have implemented functions in combination with guidance[2] and haven’t had a hiccup yet! I have a JSON parser function there, just in case, but it seems to be working reliably. Here’s a bit more of a description of using the functions API for JSON returns: https://yonom.substack.com/p/native-json-output-from-gpt-4 [0] https…
I do the same, just tell Openai to call a parser at the end and wahal.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#77One potential drawback I can see is if the viable tokens are far down the list of predictions. In that case, filtering down to just those tokens is a distribution shift with resulting output being less stable / less sensible.
It can't be less sensible JSON than syntactically invalid JSON. All the tokens higher on the list are syntax errors.
That depends highly on the values contained within the JSON. Syntactically correct is only useful if the rest of the content is useful.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#78One potential drawback I can see is if the viable tokens are far down the list of predictions. In that case, filtering down to just those tokens is a distribution shift with resulting output being less stable / less sensible.
It can't be less sensible JSON than syntactically invalid JSON. All the tokens higher on the list are syntax errors.
It seems unlikely for JSON, but this might indicate that the model has somehow painted itself into a corner and the best thing to do is backtrack?
Regenerating the entire response could be seen as an extreme form of backtracking.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#79it still blows my mind that OpenAI exposes an API with Functions calling, and yet does not guarantee the model will call your function correctly , in fact, it does not even guarantee the output will be valid JSON. When this is, really, a solved problem. I've been using github.com/microsoft/guidance for weeks, and it genuinely, truly guarantees correct output, because it simply does not sample from tokens that would b…
IANA{LLM}, but if you're only sampling from a "correct" grammar, you are potentially (very potentially) forgoing what might otherwise have been a more desirable and more semantically useful token. Most of the models have been trained on myriads of human language, not structured data necessarily, and so I'd rather elect for a more semantically enriched format (e.g. XML or YAML) because those are designed to be ~more human readable. Or perhaps more preferably: have the boss LLM pump out what it excels at (strings of prose most of the time) and have a secondary model with a stricter grammar convert that to JSON.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#80Mechanistically, I think this library takes the simple idea of masking part of the vocabulary space and steps in time efficiently. Great! I am curious, however, for the ones who have played around with such libraries wrapping base LLMs with output structure: do base models like Llama2 work very well? My experience says "hell no!" and you do need a fair bit of instruction-tuning for specific use cases to actually get…
I'm not sure of why you would want to use raw llama-2 though when there is a million super strong instruction fine-tuned versions of llama-2 on HF hub that would do the job a million times better? Like Stability-AI's Beluga-2. See https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
About your second point, the goal is that the model can only generate JSON (for example), which can 100% be done by constraining which output token can and cannot be used.