Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: LLMs can generate valid JSON 100% of the time

github.com

Post reply on HN
Print view View original

211–220 of 315 posts

Re: Show HN: LLMs can generate valid JSON 100% of the time

#211

Post by ipaddr »

andreygrehov wrote: post #187
panarky wrote: post #8

I can make GPT4 return valid JSON simply by providing examples in the system message. This works nine times out of ten. But it's still probabilistic, and nine times out of ten isn't good enough. Occasionally it will hallucinate responses like this: {"key1": "value1", "key2": "value2" for i in range(n)} Re-prompting with the parsing error message is usually enough to get it on the second try. But escaping double-quote…

Meh... I asked GPT4 to return a sample PHP code inside of a random JSON. It failed the JSON linter from the very first try. I actually couldn't pass the validation despite many retries, eg follow up corrections. Not a single time it generated a 100% valid JSON, I eventually gave up.

This worked with chatGPT: create a sample hello world in php

store that code in a json[object

code: { "php_code": "" }

Re: Show HN: LLMs can generate valid JSON 100% of the time

#213

Post by kristjansson »

dvt wrote: post #120

[flagged]

I think you might be over-simplifying. This (and llama.cpp's grammar-based sampling, which this is moving towards[1]) doesn't say "no, not like that, give me another token". It excludes impossible tokens at each step, but otherwise samples like normal.

Is this a revolutionary trick? Not really, since llama.cpp and guidance, and probably others have already done it. But it's a good trick, and hopefully one of many to justify the valuation :).

[1]: https://github.com/normal-computing/outlines/pull/178

Re: Show HN: LLMs can generate valid JSON 100% of the time

#214

Post by coder543 »

swyx wrote: post #209
coder543 wrote: post #125

As a more general comment, the repo README provides examples that all use gpt2. It would be nice to see at least one example that invokes llama2, since I feel like that would make sure the reader knows that this library can use models that are more modern and interesting.

it would also be nice to see one example that uses gpt4.

Given how this works, I don’t think that is possible unless OpenAI implements it themselves.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#216

Post by karmasimida »

panarky wrote: post #8

I can make GPT4 return valid JSON simply by providing examples in the system message. This works nine times out of ten. But it's still probabilistic, and nine times out of ten isn't good enough. Occasionally it will hallucinate responses like this: {"key1": "value1", "key2": "value2" for i in range(n)} Re-prompting with the parsing error message is usually enough to get it on the second try. But escaping double-quote…

I see grammar constrained generation for 2 major advantages:

1. It consumes fewer tokens, no need to add too many examples into the prompt.

2. It suffers less from the forgetting issue.

Another minor advantage is you can control precisely where your desired output to begin with.

But overall, those are nice perks not too substantial IMO.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#217

Post by malft »

Regex-constrained GPT, what is a mnemonic for pi?

> It's a word, a short statement or phrase which you learn.

Can you make a good one?

> Man, I wish I could recommend an answer. You're not gonna remember something, because, obviously, pi's so big. Actually, let's forget pi. There's only one way: Googling for it.

(count the letters)

Re: Show HN: LLMs can generate valid JSON 100% of the time

#220

Post by LakshyAAAgrawal »

activatedgeek wrote: post #65

Mechanistically, I think this library takes the simple idea of masking part of the vocabulary space and steps in time efficiently. Great! I am curious, however, for the ones who have played around with such libraries wrapping base LLMs with output structure: do base models like Llama2 work very well? My experience says "hell no!" and you do need a fair bit of instruction-tuning for specific use cases to actually get…

In our experience, at least for code generation, the experience has been that base models can be improved significantly by guiding token level generation.

In our paper titled "Guiding Language Models of Code with Global Context using Monitors" (https://arxiv.org/abs/2306.10763), we propose Monitor Guided Decoding, which interfaces LLMs to static analysis, and guides the model to generate type-consistent code. Without any kind of fine-tuning, we show that using static analysis to guide token level generation at specific points leads to significantly improved quality of generated code, both in terms of compilability and match with ground truth. Even very small models (1.1B) are able to generate more compilable code than much larger models (175B) while also improving on match with ground truth.

Post reply on HN