Very cool! How much latency does it add?
Show HN: LLMs can generate valid JSON 100% of the time
31–40 of 315 posts
Re: Show HN: LLMs can generate valid JSON 100% of the time
#32One potential drawback I can see is if the viable tokens are far down the list of predictions. In that case, filtering down to just those tokens is a distribution shift with resulting output being less stable / less sensible.
Indeed, this remains an empirical question.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#33it still blows my mind that OpenAI exposes an API with Functions calling, and yet does not guarantee the model will call your function correctly , in fact, it does not even guarantee the output will be valid JSON. When this is, really, a solved problem. I've been using github.com/microsoft/guidance for weeks, and it genuinely, truly guarantees correct output, because it simply does not sample from tokens that would b…
I just left a comment along these lines, but realistically it's probably cheaper to just re-emit than to add the machinery that enables this to their existing architecture. At most I could have seen them maybe running a schema validator against the output and re-requesting on your behalf, but even that's probably cheaper for them to do client side (I will say, I'm surprised their API wrapper hasn't been updated to do…
> maybe running a schema validator against the output and re-requesting on your behalf
this is the part that blows my mind. You don't have to do this! You don't have to sample the entire output, and then validate after the fact.
You're not required to greedily pick the token with the highest score. You get the scores of all tokens, on every forward pass! So why even waste time picking invalid tokens if you're just going to validate and retry later on??
(note: when I say "you" here, I mean whoever is hosting the model. It is true that OpenAI does not expose all token scores, it only gives you back the highest-scoring one. So a client-side library is not able to perform this grammar-based sampling.
BUT, OpenAI themselves host host the model, and they see all token outputs, with all scores. And in the same API request, they allow you to pass the "function definition" as a JSON schema. So why not simply apply that function definition as a mask on the token outputs? They could do this without exposing all token scores to you, which they seem very opposed to for some reason.)
Re: Show HN: LLMs can generate valid JSON 100% of the time
#34Relevant; LLama.cpp implemented grammar-based sampling last month. https://news.ycombinator.com/item?id=36819906 https://github.com/ggerganov/llama.cpp/pull/1773
We can extend our approach to grammar-based sampling, as explained in the paper linked above. Relevant PR: https://github.com/normal-computing/outlines/pull/178 Our method is much more efficient. llama.cpp loops over the entire vocabulary (~50k tokens) at each step to generate the mask. We generate an index at initialization, and building the masks at each step only requires a dictionary lookup (trade speed for memor…
It should hopefully be a quick change to llama.cpp to add a mask per grammar state to bring it in line with your generation method; I don't think the two are incompatible, thankfully.
I do wonder how much you win here by masking the tokens? You still need to iterate along the output vector to apply the mask. Masking on the accelerator still requires filtering on the CPU side? Compared to running the language model, the cost of iterating over the edges in the grammar seems small.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#35Very cool! How much latency does it add?
With our indexing approach, it only costs a dictionary lookup to get the next valid tokens during each sampling step, so very little latency.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#36OpenAI has this capability built in with functions[0], I believe! Building my own project[1] I have implemented functions in combination with guidance[2] and haven’t had a hiccup yet! I have a JSON parser function there, just in case, but it seems to be working reliably. Here’s a bit more of a description of using the functions API for JSON returns: https://yonom.substack.com/p/native-json-output-from-gpt-4 [0] https…
>OpenAI has this capability built in with functions From OpenAI's docs: > note: the model may generate invalid JSON I would guess they don't use your method - and perhaps they should!
Good catch! It really is a combination of guidance guaranteeing JSON output and OpenAI getting it right a good majority of the time[0]. But yeah, I can see how it can be frustrating that the JSON output is not guaranteed by the docs.
[0] >>99% in my experience
Re: Show HN: LLMs can generate valid JSON 100% of the time
#37Looks interesting! How would you say it compares to Microsoft's TypeChat (beyond the obvious Python/TypeScript difference)? https://microsoft.github.io/TypeChat/blog/introducing-typech...
TypeChat: let's try really hard to try to convince the model to make the highest-scoring tokens follow the grammar we want. Guidance (and this project?): Let's not even bother with trying to convince the model; instead, we'll only sample from the set of tokens that are guaranteed to be correct for the grammar we want to emit.
Yeah, and our addition to all that is to almost completely remove the cost of determining the next valid tokens on each step.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#38Earlier quoted context omitted.
>OpenAI has this capability built in with functions From OpenAI's docs: > note: the model may generate invalid JSON I would guess they don't use your method - and perhaps they should!
Good catch! It really is a combination of guidance guaranteeing JSON output and OpenAI getting it right a good majority of the time[0]. But yeah, I can see how it can be frustrating that the JSON output is not guaranteed by the docs. [0] >>99% in my experience
That said, I am definitely going to look into this library and compare its results to guidance, since they claim it blows it out of the water (which is very enticing!)
Re: Show HN: LLMs can generate valid JSON 100% of the time
#39Earlier quoted context omitted.
I just left a comment along these lines, but realistically it's probably cheaper to just re-emit than to add the machinery that enables this to their existing architecture. At most I could have seen them maybe running a schema validator against the output and re-requesting on your behalf, but even that's probably cheaper for them to do client side (I will say, I'm surprised their API wrapper hasn't been updated to do…
> maybe running a schema validator against the output and re-requesting on your behalf this is the part that blows my mind. You don't have to do this! You don't have to sample the entire output, and then validate after the fact. You're not required to greedily pick the token with the highest score. You get the scores of all tokens, on every forward pass! So why even waste time picking invalid tokens if you're just go…
Maybe re-read what I said?
> realistically it's probably cheaper to just re-emit than to add the machinery that enables this to their existing architecture
There are literally dozens of random projects that have implemented logit based masking, it's a trivial thing to implement.
What's probably not as trivial is deploying it at scale with whatever architecture OpenAI already has in place. Especially if they're using the router-based MoE architecture most people are assuming they use.
OpenAI doesn't expose token probabilities for their RLHF models, yet they did for GPT-3. Originally that lead to speculation that was to make building competitors harder, but they've now said they're actually still working on it... which leans even further into the idea they may have an architecture that makes the kind of sampling these projects rely on more difficult to implement than normal.
Given how fast and cheap they've made access to these models, their current approach is a practical workaround if that's the case.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#40One potential drawback I can see is if the viable tokens are far down the list of predictions. In that case, filtering down to just those tokens is a distribution shift with resulting output being less stable / less sensible.
It can't be less sensible JSON than syntactically invalid JSON. All the tokens higher on the list are syntax errors.