Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: LLMs can generate valid JSON 100% of the time

github.com

Post reply on HN
Print view View original

191–200 of 315 posts

Re: Show HN: LLMs can generate valid JSON 100% of the time

#191

Post by BoorishBears »

reissbaker wrote: post #134
caesil wrote: post #104

Earlier quoted context omitted.

With ChatGPT function calling I get valid JSON 100% of the time from GPT-4 unless I have made some error in prompting. The chief error is not providing escape hatches. LLMs look for a right answer. If you are feeding it some texts and asking it to return structured data about the texts, but then one of the texts is blank, it will be difficult to determine a right answer, so you get hallucinations. The solution is an…

GPT-4 is amazing, but the upside of smaller models is much lower cost. I get basically 100% accuracy on JSON modeling with GPT-4 with function calling too, but I will say that gpt-3.5-turbo with function calling is somewhat less accurate — it usually generates valid JSON in terms of JSON.parse not exploding, but not necessarily JSON following the schema I passed in (although it's surprisingly good, maybe ~90% accurat…

claude-1.2-instant came out last week and is doing extremely well at following schemas.

I'd say it's reached 3.5 turbo with the format following skills of GPT-4, which is powerful once you give it chain-of-thought

Re: Show HN: LLMs can generate valid JSON 100% of the time

#192

Post by AtlasBarfed »

Ok so:

- for what energy/processing cost per validation?

- how much of the input space was tested (unicode chars, escaped chars, newlines, etc)?

- are you doing this as a service? We've seen LLMs already evolve negatively in some capabilities over time, so do you have a constant "ping" test suite validating the LLM's performance?

Re: Show HN: LLMs can generate valid JSON 100% of the time

#194

Post by simonw »

activatedgeek wrote: post #65

Mechanistically, I think this library takes the simple idea of masking part of the vocabulary space and steps in time efficiently. Great! I am curious, however, for the ones who have played around with such libraries wrapping base LLMs with output structure: do base models like Llama2 work very well? My experience says "hell no!" and you do need a fair bit of instruction-tuning for specific use cases to actually get…

I'm quite impressed with Llama 2 13B - the more time I spend with it the more I think it might be genuinely useful for more than just playing around with local LLMs.

I'm using the MLC version (since that works with a GPU on my M2 Mac) via my https://github.com/simonw/llm-mlc plugin.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#196

Post by spockz »

ethbr1 wrote: post #107
activatedgeek wrote: post #65

Mechanistically, I think this library takes the simple idea of masking part of the vocabulary space and steps in time efficiently. Great! I am curious, however, for the ones who have played around with such libraries wrapping base LLMs with output structure: do base models like Llama2 work very well? My experience says "hell no!" and you do need a fair bit of instruction-tuning for specific use cases to actually get…

> ...given an instruction-tuned model, post-hoc masking of the state-space during generation then amounts to just changing the generation distribution... Isn't that what we did with test driven development? The primary difference was our generator functions were human instead of LLM. Why not cut out the middle-human?

Yes. And if that human was smart and knowledgable they would use property based testing to automatically generate test inputs. Most libraries make it trivial to do for custom data types and can even reduce the failing test case to a minimal size input. I have been using this since 2008 and it was around before that.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#197

Post by offmycloud »

empath-nirvana wrote: post #57
Animats wrote: post #25

OK, you get syntactically valid JSON, but does it contain the correct info? This is effectively a polisher, like spell check, which gives the output superficially correct form but doesn't understand the content. Right?

This isn't really an interesting question is it? Everyone knows that chatgpt is not an oracle. It doesn't need to output the correct information 100% of the time.

I don't think that everyone, or even a majority of people understand this. That's certainly not how AI is being marketed to the general public. The concern here is that syntactic correctness might be mistaken for factual accuracy.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#198

Post by Ycros »

Having played around with this sort of thing in the llama.cpp ecosystem when they added it a few weeks ago, I will say that it also helps if your models a) are tuned to output json and b) you prompt them to do so. Anything you can do to help the output fit the grammar helps.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#199

Post by potatoman22 »

remilouf wrote: post #16
1wheel wrote: post #9

Earlier quoted context omitted.

There's also https://lmql.ai/

LQML (and guidance https://github.com/guidance-ai/guidance ) are much more inefficient. They loop over the entire vocabulary at each step, we only do it once at initialization.

Does looping over the vocabulary add much overhead to the tok/s? I imagine they're just checking if the input is in a set, and usually there's only ~30k tokens. That's somewhat intensive, but inference on the neural net feels like it'd take longer.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#200

Post by potatoman22 »

pshc wrote: post #152
Scene_Cast2 wrote: post #23

One potential drawback I can see is if the viable tokens are far down the list of predictions. In that case, filtering down to just those tokens is a distribution shift with resulting output being less stable / less sensible.

Exactly my concern. If the model isn't sure-footed about the path forward, it seems prudent to take that fact as information and adjust the initial conditions, rather than forcing the model into a potentially hallucinatory idea-space.

What are characteristics of a "hallucinatory idea-space"? If you're enforcing the model outputting a closing bracket instead of a random string of numbers, that seems like a win for JSON formatting.
Post reply on HN