Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: LLMs can generate valid JSON 100% of the time

github.com

Post reply on HN
Print view View original

101–110 of 315 posts

Re: Show HN: LLMs can generate valid JSON 100% of the time

#101

Post by padolsey »

gowld wrote: post #98
padolsey wrote: post #64

Earlier quoted context omitted.

I've had more luck with getting it to output XML as (1) You can imbue XML with actual language/meaning (which LLMs adore) and (2) parsers can be made to be more forgiving. I get why people want to make JSON, but to me it's a bit like trying to get a cat to swim - you might eventually succeed, but it's not their natural inclination.

How do you imbue XML with meaning?

XML Elements themselves: their naming, their attributes, comments, indentation. There's more opportunity at every level of the hierarchy to demarkate and establish meaning. Having closing-tags as well, I've found, is a massive boon; LLMs can better understand what "finishing" looks like if its delimited in a semantic way - with a name.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#102

Post by skybrian »

chipsrafferty wrote: post #81
coder543 wrote: post #44

Earlier quoted context omitted.

This analogy falls apart because the spellchecker is separate from the author, and doesn’t know what the author intended. Here, the LLM is still dictating the token probabilities, so the content will be as correct as the LLM can make it, given the constraints. AIUI, the sampler is just choosing tokens on a combination of probability and syntactic correctness, instead of strictly on probability. If the LLM is forced t…

Why isn't it possible to design LLMs that say "I don't know"?

They do, but it's a form of imitation, not actually knowing what they don't know.

Ask an LLM to imitate a confident physicist and it will try, regardless of how much physics it knows.

Or if you tell ChatGPT that it's wrong multiple times, it may learn the pattern and assume it's always wrong, resulting in a downward spiral. (This can happen when using Code Interpreter and it makes several failed attempts to correct a mistake.)

The difficult research problem is training it to have an accurate model of what it knows.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#103

Post by simonw »

I really hope OpenAI add something like this to their endpoints soon.

Being able to pass up some kind of grammar (a regular expression, or a JSON schema, or some other format) and have this trick run during their token sampling process to ensure the output was compliant would be incredibly useful.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#104

Post by caesil »

panarky wrote: post #8

I can make GPT4 return valid JSON simply by providing examples in the system message. This works nine times out of ten. But it's still probabilistic, and nine times out of ten isn't good enough. Occasionally it will hallucinate responses like this: {"key1": "value1", "key2": "value2" for i in range(n)} Re-prompting with the parsing error message is usually enough to get it on the second try. But escaping double-quote…

With ChatGPT function calling I get valid JSON 100% of the time from GPT-4 unless I have made some error in prompting.

The chief error is not providing escape hatches. LLMs look for a right answer. If you are feeding it some texts and asking it to return structured data about the texts, but then one of the texts is blank, it will be difficult to determine a right answer, so you get hallucinations. The solution is an escape hatch where one of the arguments is a `textIsMissing` boolean or something.

As long as you've accounted for these failure modes, it works flawlessly.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#105

Post by burke »

Animats wrote: post #25

OK, you get syntactically valid JSON, but does it contain the correct info? This is effectively a polisher, like spell check, which gives the output superficially correct form but doesn't understand the content. Right?

You can go pretty deep once you get context free grammars. For example, I'm using torch-grammar (but outlines should be able to do the same thing once CFG support is merged) to not just restrict the format of a generation to a DSL's syntax, but to restrict the keys it updates to valid keys in a known set.

e.g.:

    int_key ::= DQUO ("f" ("e" ("atured-" ("b" ("log." ("p" ("ost_limit" | "a" ...
Obviously, yeah, it doesn't "understand" the content, but that's what the LLM is for. It's remarkable how plausible the generations you can get out of random noise are with a sufficiently-restrictive grammar. Bolting that onto a well-trained LLM is pretty powerful.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#106

Post by baobabKoodaa »

Kiro wrote: post #69

Does this mean that I need to call the LLM API once for each token?

No. You need to hook into the LLM at a lower level. One API call typically triggers a generation of a sequence of tokens and this library has to poke into things between each generated token.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#107

Post by ethbr1 »

activatedgeek wrote: post #65

Mechanistically, I think this library takes the simple idea of masking part of the vocabulary space and steps in time efficiently. Great! I am curious, however, for the ones who have played around with such libraries wrapping base LLMs with output structure: do base models like Llama2 work very well? My experience says "hell no!" and you do need a fair bit of instruction-tuning for specific use cases to actually get…

> ...given an instruction-tuned model, post-hoc masking of the state-space during generation then amounts to just changing the generation distribution...

Isn't that what we did with test driven development?

The primary difference was our generator functions were human instead of LLM. Why not cut out the middle-human?

Re: Show HN: LLMs can generate valid JSON 100% of the time

#108

Post by huevosabio »

nabakin wrote: post #89
make3 wrote: post #80

Earlier quoted context omitted.

I'm not sure of why you would want to use raw llama-2 though when there is a million super strong instruction fine-tuned versions of llama-2 on HF hub that would do the job a million times better? Like Stability-AI's Beluga-2. See https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... About your second point, the goal is that the model can only generate JSON (for example), which can 100% be done by constrain…

Don't rely too much on automated benchmarks for LLMs. They are often gamed, made to overfit, and result in worse performance in the general case. Human evaluation is the gold standard and the Llama 2 paper gave significant evidence that Llama 2 70b chat is on-par, if not, better than ChatGPT for that metric so I tend to stick to it unless there is good reason not to.

The problem with Llama 2 chat versions is that they have been RLHF-ed to death. You can't ask questions without getting a sermon of how your question may be inappropriate for this or that reason.

I think it's worse on the smaller models, but still present in the 70B one.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#110

Post by lefttoreader »

The “trick” seems to blatantly rip off FlashText without citing it?

https://arxiv.org/pdf/1711.00046.pdf

I’m a fan of the approach. I normally wouldn’t care if this was just another LLM library taking inspiration, but if you’re going to go out of your way to put a paper on the ArXiv, feels like doing a literature review is a good step?

Post reply on HN