How does this compare in terms of latency, cost, and effectiveness to jsonformer? https://github.com/1rgs/jsonformer
jsonformer uses a template rather than a DFA. The logit masking seems to be identical, though.
Show HN: LLMs can generate valid JSON 100% of the time
51–60 of 315 posts
Re: Show HN: LLMs can generate valid JSON 100% of the time
#52"Generating valid JSON" is not impressive. Here's some valid JSON: [] The tricky part is generating useful JSON.
Generating valid JSON that conforms to a given schema is pretty useful, although not impressive by itself. If the model can deduce field values from schema alone though, I think it's pretty neat.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#53This is exciting, we built a similar tool[1] recently specifically targeted at constraining llama output to match a TypeScript interface.
I firmly believe that output format guarantees are going to be important for real (non-toy) decades for LLMs
Re: Show HN: LLMs can generate valid JSON 100% of the time
#54How does this compare in terms of latency, cost, and effectiveness to jsonformer? https://github.com/1rgs/jsonformer
Figure 2 in our paper (https://arxiv.org/abs/2307.09702) shows the difference between guidance and outlines to generate a sequence that is valid to a regex. Jsonformer uses the same technique as guidance. Extrapolate this to several fields.
Note that we still need to manage the KV cache in outlines. It’s a small interface change that will be made this week hopefully, but we’ve been focusing on constrained generation so far.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#55How does this relate to ggmls bnf sampling?
Two differences: (1) This feature only requires regex-guided generation. We have a PR for BNF sampling that is about to be merged. (2) ggml loops over the entire vocabulary (~50k tokens) at each step, which introduces a noticeable overhead, and makes it unusable for complex grammars. Our method works by building an index at initialization, and build the masks at each step with a dictionary lookup. Once the index is b…
Regex-guided gen is slick… is it arbitrary? Or are you custom building it for json?
If arbitrary, how are you pre-defining a set of masks? I would expect that splitting an arbitrary regex into a bunch of contexts for a masking dictionary to be non-trivial.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#56https://github.com/newhouseb/clownfish
Which I've been using for a while now, also restricts the sampling space to force correct generation, but does so as the result of a different process than yours.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#57OK, you get syntactically valid JSON, but does it contain the correct info? This is effectively a polisher, like spell check, which gives the output superficially correct form but doesn't understand the content. Right?
This isn't really an interesting question is it? Everyone knows that chatgpt is not an oracle. It doesn't need to output the correct information 100% of the time.
Re: Show HN: LLMs can generate valid JSON 100% of the time
#58Is this Brandon Willard the breakdancer from Detroit Brandon Willard?
Edit: It is! https://brandonwillard.github.io/
Re: Show HN: LLMs can generate valid JSON 100% of the time
#59it still blows my mind that OpenAI exposes an API with Functions calling, and yet does not guarantee the model will call your function correctly , in fact, it does not even guarantee the output will be valid JSON. When this is, really, a solved problem. I've been using github.com/microsoft/guidance for weeks, and it genuinely, truly guarantees correct output, because it simply does not sample from tokens that would b…
I think this is likely a consequence of a couple of factors:
1. Fancy token selection w/in batches (read: beam search) is probably fairly hard to implement at scale without a significant loss in GPU utilization. Normally you can batch up a bunch of parallel generations and just push them all through the LLM at once because every generated token (of similar prompt size + some padding perhaps) takes a predictable time. If you stick a parser in between every token that can take variable time then your batch is slowed by the most complex grammar of the bunch.
2. OpenAI appears to work under the thesis articulated in the Bitter Lesson [i] that more compute (either via fine-tuning or bigger models) is the least foolish way to achieve improved capabilities hence their approach of function-calling just being... a fine tuned model.
[i] http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Re: Show HN: LLMs can generate valid JSON 100% of the time
#60I also released a hosted version of my open-source libraries ReLLM and ParserLLM that already supports APIs for
* Regex completion for LLMs
* Context-free Grammar completion for LLMs
[0] https://github.com/r2d4/rellm
[1] https://github.com/r2d4/parserllm
[2] https://github.com/thiggle/api
There's also another API on Thiggle that I've build that supports classification via a similar logit-based strategy.