Is this Brandon Willard the breakdancer from Detroit Brandon Willard? Edit: It is! https://brandonwillard.github.io/
Ha, yeah, in a distant, but really fun, past!
Show HN: LLMs can generate valid JSON 100% of the time
61–70 of 315 posts
Re: Show HN: LLMs can generate valid JSON 100% of the time
#62So to explain this another way: After each token generated by the LLM you update the logit bias “mask” to only allow the next token to be a valid json token? Very slick!
You would also need to keep generating until the whole string is valid. And what if it gets caught in a loop?
Not sure how this can really guarantee 100%
Re: Show HN: LLMs can generate valid JSON 100% of the time
#63I feel like I'm missing something very basic here, but is this library intended to be used with an existing model? If so, could you point to an example?
Re: Show HN: LLMs can generate valid JSON 100% of the time
#64I can make GPT4 return valid JSON simply by providing examples in the system message. This works nine times out of ten. But it's still probabilistic, and nine times out of ten isn't good enough. Occasionally it will hallucinate responses like this: {"key1": "value1", "key2": "value2" for i in range(n)} Re-prompting with the parsing error message is usually enough to get it on the second try. But escaping double-quote…
I've had more luck with getting it to output XML as (1) You can imbue XML with actual language/meaning (which LLMs adore) and (2) parsers can be made to be more forgiving. I get why people want to make JSON, but to me it's a bit like trying to get a cat to swim - you might eventually succeed, but it's not their natural inclination.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#65Mechanistically, I think this library takes the simple idea of masking part of the vocabulary space and steps in time efficiently. Great!
I am curious, however, for the ones who have played around with such libraries wrapping base LLMs with output structure: do base models like Llama2 work very well? My experience says "hell no!" and you do need a fair bit of instruction-tuning for specific use cases to actually get things to work.
And even then, it seems very counter-intuitive to me that given an instruction-tuned model, post-hoc masking of the state-space during generation then amounts to just changing the generation distribution, and potentially detrimental to instruction-tuning?
Re: Show HN: LLMs can generate valid JSON 100% of the time
#66I also released a hosted version of my open-source libraries ReLLM and ParserLLM that already supports APIs for * Regex completion for LLMs * Context-free Grammar completion for LLMs https://thiggle.com/ [0] https://github.com/r2d4/rellm [1] https://github.com/r2d4/parserllm [2] https://github.com/thiggle/api There's also another API on Thiggle that I've build that supports classification via a similar logit-based st…
[deleted]
Re: Show HN: LLMs can generate valid JSON 100% of the time
#67I feel like I'm missing something very basic here, but is this library intended to be used with an existing model? If so, could you point to an example?
It can be used with any open source model (if you can get the logits), and to some extent with OpenAI's API. Here is an example with `transformers`: https://github.com/normal-computing/outlines#efficient-json-...
We plan on adding more model integrations, but it is completely decoupled from the method implementation.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#68How does this compare in terms of latency, cost, and effectiveness to jsonformer? https://github.com/1rgs/jsonformer
Figure 2 in our paper ( https://arxiv.org/abs/2307.09702 ) shows the difference between guidance and outlines to generate a sequence that is valid to a regex. Jsonformer uses the same technique as guidance. Extrapolate this to several fields. Note that we still need to manage the KV cache in outlines. It’s a small interface change that will be made this week hopefully, but we’ve been focusing on constrained generatio…
Sad to see that my related work on token-level constrained text generation is not cited in the paper: https://github.com/Hellisotherpeople/Constrained-Text-Genera...
Re: Show HN: LLMs can generate valid JSON 100% of the time
#69Does this mean that I need to call the LLM API once for each token?
Re: Show HN: LLMs can generate valid JSON 100% of the time
#70I can make GPT4 return valid JSON simply by providing examples in the system message. This works nine times out of ten. But it's still probabilistic, and nine times out of ten isn't good enough. Occasionally it will hallucinate responses like this: {"key1": "value1", "key2": "value2" for i in range(n)} Re-prompting with the parsing error message is usually enough to get it on the second try. But escaping double-quote…
I wonder if the next iteration of OpenAI features is something like:
right now you can inject prompts that the LLM takes into consideration before the output
I wonder if you can make it have a "post" generation function that says like "keep re-trying in a loop (aka hallucinating with randomness) until the output message passes XYZ format/checks/scoring"