Have you found a solution to output exceeding the context window? That's been our only issue with generating json output.
Show HN: LLMs can generate valid JSON 100% of the time
221–230 of 315 posts
Re: Show HN: LLMs can generate valid JSON 100% of the time
#222It does seem inapt to claim this “eliminates” hallucinations in your blog post. Sort of like unnamed FP languages claiming to eliminate bugs.
Both eliminate a subclass of failures, but don’t preclude failure categorically.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#223Earlier quoted context omitted.
I'm quite impressed with Llama 2 13B - the more time I spend with it the more I think it might be genuinely useful for more than just playing around with local LLMs. I'm using the MLC version (since that works with a GPU on my M2 Mac) via my https://github.com/simonw/llm-mlc plugin.
What are your use cases
Not simonw, but I've been using Llama2-13B for search re-ranking very successfully.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#224How is this different from generating such things without an LLM? In other words picking random valid tokens from the grammar via fuzzing or similar techniques.
Instead of downvoting, I’d appreciate an answer. I’m genuinely curious to learn what the value add of the LLM is.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#225Regex-constrained GPT, what is a mnemonic for pi? > It's a word, a short statement or phrase which you learn. Can you make a good one? > Man, I wish I could recommend an answer. You're not gonna remember something, because, obviously, pi's so big. Actually, let's forget pi. There's only one way: Googling for it. (count the letters)
[deleted]
Re: Show HN: LLMs can generate valid JSON 100% of the time
#226For complex tasks like coding, my experience is that asking for a complex output format hurts performance on the underlying task. This showed up clearly in code editing benchmarks of GPT-3.5 and GPT-4: https://aider.chat/docs/benchmarks.html I’m curious if you have measured whether the “constrained generation” that you’re doing suffers from similar downsides?
100% have observed the same over many tests. No loss in fidelity when responding in spoken language style of formatting but using json is disastrous.
While not ideal, could a workaround be to ask in spoken language first, and then ask to format it in JSON?
Re: Show HN: LLMs can generate valid JSON 100% of the time
#227It does seem inapt to claim this “eliminates” hallucinations in your blog post. Sort of like unnamed FP languages claiming to eliminate bugs. Both eliminate a subclass of failures, but don’t preclude failure categorically.
As it describes it does eliminate non JSON outputs by masking the tokens while the LLM is generating. Its quite smart if you ask me.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#228it still blows my mind that OpenAI exposes an API with Functions calling, and yet does not guarantee the model will call your function correctly , in fact, it does not even guarantee the output will be valid JSON. When this is, really, a solved problem. I've been using github.com/microsoft/guidance for weeks, and it genuinely, truly guarantees correct output, because it simply does not sample from tokens that would b…
I think this is likely a consequence of a couple of factors: 1. Fancy token selection w/in batches (read: beam search) is probably fairly hard to implement at scale without a significant loss in GPU utilization. Normally you can batch up a bunch of parallel generations and just push them all through the LLM at once because every generated token (of similar prompt size + some padding perhaps) takes a predictable time.…
The "Bitter Lesson" indeed sheds light on the future trajectory of technology, emphasizing the supremacy of computation over human-designed methods. However, our current value functions often still need to focus on what we can achieve with the tools and methods available to us today. While it's likely that computational tools will eventually replace human-guided "outlines" or "guidance", that are used to shape LLM outputs, there will likely always be a substantial amount of human-structured knobs necessary to align computation with our immediate needs and goals.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#229It does seem inapt to claim this “eliminates” hallucinations in your blog post. Sort of like unnamed FP languages claiming to eliminate bugs. Both eliminate a subclass of failures, but don’t preclude failure categorically.
As it describes it does eliminate non JSON outputs by masking the tokens while the LLM is generating. Its quite smart if you ask me.
It’s very clever. I wouldn’t want it to be oversold.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#230Mechanistically, I think this library takes the simple idea of masking part of the vocabulary space and steps in time efficiently. Great! I am curious, however, for the ones who have played around with such libraries wrapping base LLMs with output structure: do base models like Llama2 work very well? My experience says "hell no!" and you do need a fair bit of instruction-tuning for specific use cases to actually get…
In our experience, at least for code generation, the experience has been that base models can be improved significantly by guiding token level generation. In our paper titled "Guiding Language Models of Code with Global Context using Monitors" ( https://arxiv.org/abs/2306.10763 ), we propose Monitor Guided Decoding, which interfaces LLMs to static analysis, and guides the model to generate type-consistent code. Witho…
It is an interesting paper. Any idea when the code/data will be released? It appears it has been almost 2 months since the paper was submitted, but the link given leads to a random bing page :-(