Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: LLMs can generate valid JSON 100% of the time

github.com

Post reply on HN
Print view View original

151–160 of 315 posts

Re: Show HN: LLMs can generate valid JSON 100% of the time

#151

Post by nouri »

infecto wrote: post #136
anotherpaulg wrote: post #131

For complex tasks like coding, my experience is that asking for a complex output format hurts performance on the underlying task. This showed up clearly in code editing benchmarks of GPT-3.5 and GPT-4: https://aider.chat/docs/benchmarks.html I’m curious if you have measured whether the “constrained generation” that you’re doing suffers from similar downsides?

100% have observed the same over many tests. No loss in fidelity when responding in spoken language style of formatting but using json is disastrous.

Using OpenAI Function Calls or asking for JSON in the prompt?

Re: Show HN: LLMs can generate valid JSON 100% of the time

#152

Post by pshc »

Scene_Cast2 wrote: post #23

One potential drawback I can see is if the viable tokens are far down the list of predictions. In that case, filtering down to just those tokens is a distribution shift with resulting output being less stable / less sensible.

Exactly my concern. If the model isn't sure-footed about the path forward, it seems prudent to take that fact as information and adjust the initial conditions, rather than forcing the model into a potentially hallucinatory idea-space.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#153

Post by quickthrower2 »

coder543 wrote: post #145
quickthrower2 wrote: post #138

Earlier quoted context omitted.

The word "lie" is probably too anthropic here. I should have just said "made up". There is no intent to lie. And the model isn't try to self-fact-check anyway. (Maybe some do). But if they do they are probably bad at it at the moment, at least from my experience of GPT3.5 (not used 4 much).

> at least from my experience of GPT3.5 (not used 4 much). And 4 is tremendously better than 3.5, in my own experience. Not perfect, but actually useful.

Can anyone recommend a good, and trusted UI so I can use it via the API? I don't want to pay monthly for it, but would be nice to use occasionally. I keep meaning to do this!

Re: Show HN: LLMs can generate valid JSON 100% of the time

#154

Post by RyEgswuCsn »

quickthrower2 wrote: post #130

> LLMs can generate valid JSON 100% of the time If that seems surprising, it is worth doing a course like Karpathy's zero to hero NN, and have all the magic peeled away a layer at a time. The reason you can do this is because LLMs don't just generate the next word or token, it produces a probability distribution over all tokens. A JSON parser can give you a list of next valid tokens. The tokens in each case might be…

How does the LLM know what valid JSON tokens are?

What if the training data contains malformed JSON? There ought to be a non-zero chance of the LLM producing invalid JSON, no?

Re: Show HN: LLMs can generate valid JSON 100% of the time

#155

Post by jmcminis »

Are there edge cases here due to context length?

1. I have a json schema with required fields. I complete the json, but do not include the required fields.

2. I run out of token from the model before I finish the json object because I'm in the middle of some deep, nested structure.

These seem solvable, just edge cases to control for by either reserving tokens, randomly generating required tokens until completing the json, or something more sophisticated.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#156

Post by dvasdekis »

remilouf wrote: post #129
coder543 wrote: post #127

Earlier quoted context omitted.

Maybe... but then if I want to use something better, I have to figure out how by myself. I said "at least one example", not "please change all the examples to llama2." I agree with your general point. It would be nice if there were an example of how to use a better model. Models often have different shapes and requirements, so is it really as simple as changing the string "gpt2" to "llama2-13B-Chat" and it will magic…

Agree, working on a Colab with a "better" model as we speak.

Wonderful, thank you!

Re: Show HN: LLMs can generate valid JSON 100% of the time

#157

Post by activatedgeek »

ethbr1 wrote: post #107
activatedgeek wrote: post #65

Mechanistically, I think this library takes the simple idea of masking part of the vocabulary space and steps in time efficiently. Great! I am curious, however, for the ones who have played around with such libraries wrapping base LLMs with output structure: do base models like Llama2 work very well? My experience says "hell no!" and you do need a fair bit of instruction-tuning for specific use cases to actually get…

> ...given an instruction-tuned model, post-hoc masking of the state-space during generation then amounts to just changing the generation distribution... Isn't that what we did with test driven development? The primary difference was our generator functions were human instead of LLM. Why not cut out the middle-human?

I think what I am saying is tangential to TDD. I am not really even concerned about the ability of LLM to function as desired, and its verification.

I was rather concerned about a broader fundamental question - how does post-hoc guided generation interfere with the potential benefits of instruction-tuning?

Re: Show HN: LLMs can generate valid JSON 100% of the time

#158

Post by activatedgeek »

make3 wrote: post #80
activatedgeek wrote: post #65

Mechanistically, I think this library takes the simple idea of masking part of the vocabulary space and steps in time efficiently. Great! I am curious, however, for the ones who have played around with such libraries wrapping base LLMs with output structure: do base models like Llama2 work very well? My experience says "hell no!" and you do need a fair bit of instruction-tuning for specific use cases to actually get…

I'm not sure of why you would want to use raw llama-2 though when there is a million super strong instruction fine-tuned versions of llama-2 on HF hub that would do the job a million times better? Like Stability-AI's Beluga-2. See https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... About your second point, the goal is that the model can only generate JSON (for example), which can 100% be done by constrain…

> I'm not sure of why you would want to use raw llama-2

Sure. My concern was not specific to llama-2, and was only using it as a placeholder example of a decent pre-trained base model. Replace it with your favorite base model, which you want to use for guided generation. My question is more fundamental - how does post-hoc guided generation interfere with the potential benefits of instruction-tuning?

> About your second point, the goal is that the model can only generate JSON (for example), which can 100% be done by constraining which output token can and cannot be used.

Mechanistically, yes. I am not arguing that. The whole point is to generate JSON that is "useful".

Re: Show HN: LLMs can generate valid JSON 100% of the time

#160

Post by OJFord »

quickthrower2 wrote: post #130

> LLMs can generate valid JSON 100% of the time If that seems surprising, it is worth doing a course like Karpathy's zero to hero NN, and have all the magic peeled away a layer at a time. The reason you can do this is because LLMs don't just generate the next word or token, it produces a probability distribution over all tokens. A JSON parser can give you a list of next valid tokens. The tokens in each case might be…

Maybe it's just me, but I'm not doing anything that calls itself 'zero to hero'. Would love some good resources (preferably textbook, or at least written) on LLMs though. I don't even understand the link to 'generative' image/video AI, which seems to have exploded at roughly the same time and surely isn't a coincidence.

I studied a little (literally 'intro to') ML at university, about enough to grok it as an application of stats, tie into things seen elsewhere, but not really more than that.

Every supposéd tutorial or explainer I've seen posted here or been able to find has been a weird (IMO) mix if simultaneously assuming a decent (at least greater than mine) ML background, but also really dumbed down clone this repo download that model switch between them like this, fine-tune them by cd'ing to this directory and ... Ok but what's actually going on?

Post reply on HN