Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: LLMs can generate valid JSON 100% of the time

github.com

Post reply on HN
Print view View original

201–210 of 315 posts

Re: Show HN: LLMs can generate valid JSON 100% of the time

#201

Post by nabakin »

huevosabio wrote: post #108
nabakin wrote: post #89

Earlier quoted context omitted.

Don't rely too much on automated benchmarks for LLMs. They are often gamed, made to overfit, and result in worse performance in the general case. Human evaluation is the gold standard and the Llama 2 paper gave significant evidence that Llama 2 70b chat is on-par, if not, better than ChatGPT for that metric so I tend to stick to it unless there is good reason not to.

The problem with Llama 2 chat versions is that they have been RLHF-ed to death. You can't ask questions without getting a sermon of how your question may be inappropriate for this or that reason. I think it's worse on the smaller models, but still present in the 70B one.

For sure, that's a good reason for using the uncensored fine-tuned versions. There are other good reasons too like expanded context size, codegen, and story writing/rp. Just be careful of extraordinary benchmarks.

Btw, have you tried changing the default Llama 2 chat prompt? Meta tried to fine-tune it so that if you remove the safety part from the prompt, safety won't be applied[1]. Not sure how well it works myself, but worth a shot I guess

[1] can be found in the Llama 2 paper

Re: Show HN: LLMs can generate valid JSON 100% of the time

#202

Post by moneywoes »

simonw wrote: post #194
activatedgeek wrote: post #65

Mechanistically, I think this library takes the simple idea of masking part of the vocabulary space and steps in time efficiently. Great! I am curious, however, for the ones who have played around with such libraries wrapping base LLMs with output structure: do base models like Llama2 work very well? My experience says "hell no!" and you do need a fair bit of instruction-tuning for specific use cases to actually get…

I'm quite impressed with Llama 2 13B - the more time I spend with it the more I think it might be genuinely useful for more than just playing around with local LLMs. I'm using the MLC version (since that works with a GPU on my M2 Mac) via my https://github.com/simonw/llm-mlc plugin.

What are your use cases

Re: Show HN: LLMs can generate valid JSON 100% of the time

#203

Post by prempv »

padolsey wrote: post #64
panarky wrote: post #8

I can make GPT4 return valid JSON simply by providing examples in the system message. This works nine times out of ten. But it's still probabilistic, and nine times out of ten isn't good enough. Occasionally it will hallucinate responses like this: {"key1": "value1", "key2": "value2" for i in range(n)} Re-prompting with the parsing error message is usually enough to get it on the second try. But escaping double-quote…

I've had more luck with getting it to output XML as (1) You can imbue XML with actual language/meaning (which LLMs adore) and (2) parsers can be made to be more forgiving. I get why people want to make JSON, but to me it's a bit like trying to get a cat to swim - you might eventually succeed, but it's not their natural inclination.

I've had the same experience as well. I suspect if it's due to large presence of HTML in the training data as part of codebases and online content

Re: Show HN: LLMs can generate valid JSON 100% of the time

#204

Post by kristjansson »

dontreact wrote: post #62
J_Shelby_J wrote: post #15

So to explain this another way: After each token generated by the LLM you update the logit bias “mask” to only allow the next token to be a valid json token? Very slick!

You would also need to keep generating until the whole string is valid. And what if it gets caught in a loop? Not sure how this can really guarantee 100%

Same problem with normal sampling - if it doesn't pick the token, you're stuck generating until you hit some stopping heuristic (max tokens, timeout, etc.)

Re: Show HN: LLMs can generate valid JSON 100% of the time

#205

Post by simonw »

moneywoes wrote: post #202
simonw wrote: post #194

Earlier quoted context omitted.

I'm quite impressed with Llama 2 13B - the more time I spend with it the more I think it might be genuinely useful for more than just playing around with local LLMs. I'm using the MLC version (since that works with a GPU on my M2 Mac) via my https://github.com/simonw/llm-mlc plugin.

What are your use cases

The thing I really want to get working is retrieval augmented generation - so effectively answering questions based on a blob of context that I pass in, and being able to do good-enough summarization.

I haven't quite proved this to myself yet but I think it's going to work pretty well.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#206

Post by kristjansson »

MuffinFlavored wrote: post #70
panarky wrote: post #8

I can make GPT4 return valid JSON simply by providing examples in the system message. This works nine times out of ten. But it's still probabilistic, and nine times out of ten isn't good enough. Occasionally it will hallucinate responses like this: {"key1": "value1", "key2": "value2" for i in range(n)} Re-prompting with the parsing error message is usually enough to get it on the second try. But escaping double-quote…

I wonder if the next iteration of OpenAI features is something like: right now you can inject prompts that the LLM takes into consideration before the output I wonder if you can make it have a "post" generation function that says like "keep re-trying in a loop (aka hallucinating with randomness) until the output message passes XYZ format/checks/scoring"

Why wait for OpenAI?

Re: Show HN: LLMs can generate valid JSON 100% of the time

#207

Post by ipaddr »

selcuka wrote: post #172
caesil wrote: post #104

Earlier quoted context omitted.

With ChatGPT function calling I get valid JSON 100% of the time from GPT-4 unless I have made some error in prompting. The chief error is not providing escape hatches. LLMs look for a right answer. If you are feeding it some texts and asking it to return structured data about the texts, but then one of the texts is blank, it will be difficult to determine a right answer, so you get hallucinations. The solution is an…

The premise of function calling is great, but in my experience (at least on GPT-3.5, haven't tried it with GPT-4 yet) it seems to generate wildly different, and less useful results, for the same prompt.

You can change the randomness value to 0 and get the same output each time for the same text

Re: Show HN: LLMs can generate valid JSON 100% of the time

#209

Post by swyx »

coder543 wrote: post #125

As a more general comment, the repo README provides examples that all use gpt2. It would be nice to see at least one example that invokes llama2, since I feel like that would make sure the reader knows that this library can use models that are more modern and interesting.

it would also be nice to see one example that uses gpt4.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#210

Post by selcuka »

ipaddr wrote: post #207
selcuka wrote: post #172

Earlier quoted context omitted.

The premise of function calling is great, but in my experience (at least on GPT-3.5, haven't tried it with GPT-4 yet) it seems to generate wildly different, and less useful results, for the same prompt.

You can change the randomness value to 0 and get the same output each time for the same text

I should probably re-test it, but I think it wasn't the temperature. The results were unusually useless.
Post reply on HN