Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: LLMs can generate valid JSON 100% of the time

github.com

Post reply on HN
Print view View original

161–170 of 315 posts

Re: Show HN: LLMs can generate valid JSON 100% of the time

#161

Post by catlifeonmars »

RyEgswuCsn wrote: post #154
quickthrower2 wrote: post #130

> LLMs can generate valid JSON 100% of the time If that seems surprising, it is worth doing a course like Karpathy's zero to hero NN, and have all the magic peeled away a layer at a time. The reason you can do this is because LLMs don't just generate the next word or token, it produces a probability distribution over all tokens. A JSON parser can give you a list of next valid tokens. The tokens in each case might be…

How does the LLM know what valid JSON tokens are? What if the training data contains malformed JSON? There ought to be a non-zero chance of the LLM producing invalid JSON, no?

I think the idea is that it’s easy to filter the result set to restrict to just valid JSON.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#162

Post by mr_toad »

chipsrafferty wrote: post #81
coder543 wrote: post #44

Earlier quoted context omitted.

This analogy falls apart because the spellchecker is separate from the author, and doesn’t know what the author intended. Here, the LLM is still dictating the token probabilities, so the content will be as correct as the LLM can make it, given the constraints. AIUI, the sampler is just choosing tokens on a combination of probability and syntactic correctness, instead of strictly on probability. If the LLM is forced t…

Why isn't it possible to design LLMs that say "I don't know"?

> Why isn't it possible to design LLMs that say "I don't know"?

You have to have an understanding of ‘I’ before you can make that judgement.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#163

Post by MereInterest »

RyEgswuCsn wrote: post #154
quickthrower2 wrote: post #130

> LLMs can generate valid JSON 100% of the time If that seems surprising, it is worth doing a course like Karpathy's zero to hero NN, and have all the magic peeled away a layer at a time. The reason you can do this is because LLMs don't just generate the next word or token, it produces a probability distribution over all tokens. A JSON parser can give you a list of next valid tokens. The tokens in each case might be…

How does the LLM know what valid JSON tokens are? What if the training data contains malformed JSON? There ought to be a non-zero chance of the LLM producing invalid JSON, no?

LLMs work by outputting a value for each token, then using those values to generate a probability distribution. Usually, this will be through a function like softmax [0], but there's nothing preventing you from doing some post-processing first. That post processing could be aware of the tokens that would be valid as the next token in a JSON format, and set the probabilities of all other tokens to zero. That way, even if the training data contains malformed JSON, the generator is still constrained to produce valid JSON.

[0] https://en.wikipedia.org/wiki/Softmax_function

Re: Show HN: LLMs can generate valid JSON 100% of the time

#164

Post by reasonabl_human »

newhouseb wrote: post #59
2bitencryption wrote: post #27

it still blows my mind that OpenAI exposes an API with Functions calling, and yet does not guarantee the model will call your function correctly , in fact, it does not even guarantee the output will be valid JSON. When this is, really, a solved problem. I've been using github.com/microsoft/guidance for weeks, and it genuinely, truly guarantees correct output, because it simply does not sample from tokens that would b…

I think this is likely a consequence of a couple of factors: 1. Fancy token selection w/in batches (read: beam search) is probably fairly hard to implement at scale without a significant loss in GPU utilization. Normally you can batch up a bunch of parallel generations and just push them all through the LLM at once because every generated token (of similar prompt size + some padding perhaps) takes a predictable time.…

What a fascinating read, thanks for sharing that link.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#165

Post by lefttoreader »

verdverm wrote: post #149
lefttoreader wrote: post #110

The “trick” seems to blatantly rip off FlashText without citing it? https://arxiv.org/pdf/1711.00046.pdf I’m a fan of the approach. I normally wouldn’t care if this was just another LLM library taking inspiration, but if you’re going to go out of your way to put a paper on the ArXiv, feels like doing a literature review is a good step?

Care to explain how a string replacement algorithm relates to nudging the logits of a ML model? I don't see the "rip off", the paper you cite requires a complete document to work on while this work is for guiding the generation of tokens

Sure! So it’s hopefully clear that the notion of constrained grammar is not novel (see every comment on here of people name-dropping their implementation from two months ago).

The novelty here is “instead of checking whether every token is allowed” to create a finite state machine that defines which tokens are allowable at each generation step. This lets them not check every token at every step.

The trick of creating an FSM to efficiently check next-token grammar is what allowed FlashText to run circles around standard regex stuff. Even FlashText guy acknowledged the shoulders he stood on, etc.

Let’s be super clear here, none of these standards apply when you’re building good ole libraries. But putting out a paper really elevates what you’re on the hook for. Most folks that write papers are dying to acknowledge the shoulders they stand on - it’s part of the toxic humility we all engage in.

Again - shill OSS all day - I’ll upvote it.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#166

Post by bhickey »

verdverm wrote: post #149
lefttoreader wrote: post #110

The “trick” seems to blatantly rip off FlashText without citing it? https://arxiv.org/pdf/1711.00046.pdf I’m a fan of the approach. I normally wouldn’t care if this was just another LLM library taking inspiration, but if you’re going to go out of your way to put a paper on the ArXiv, feels like doing a literature review is a good step?

Care to explain how a string replacement algorithm relates to nudging the logits of a ML model? I don't see the "rip off", the paper you cite requires a complete document to work on while this work is for guiding the generation of tokens

Both papers use the phrase "regular expressions" and there the resemblance ends. The linked manuscript uses regular expression to realize a grammar and then memoizes logic masks. I want to know why FlashText failed to cite:

Baeza-Yates, Ricardo A., and Gaston H. Gonnet. "Fast text searching for regular expressions or automaton searching on tries." Journal of the ACM (JACM) 43.6 (1996): 915-936.

Eltabakh, Mohamed Y., Ramy Eltarras, and Walid G. Aref. "To trie or not to trie? realizing space-partitioning trees inside postgresql: Challenges, experiences and performance." (2005).

Zhang, Yijun, and Lizhen Xu. "An algorithm for url routing based on trie structure." 2015 12th Web Information System and Application Conference (WISA). IEEE, 2015.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#167

Post by catlifeonmars »

padjo wrote: post #97
MuffinFlavored wrote: post #70

Earlier quoted context omitted.

I wonder if the next iteration of OpenAI features is something like: right now you can inject prompts that the LLM takes into consideration before the output I wonder if you can make it have a "post" generation function that says like "keep re-trying in a loop (aka hallucinating with randomness) until the output message passes XYZ format/checks/scoring"

It’s starting to feel like LLMs are to “classical” software engineering what quantum physics was to classical physics

How so? I’m not quite following the analogy.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#168

Post by neoncontrails »

dwattttt wrote: post #139
quickthrower2 wrote: post #130

> LLMs can generate valid JSON 100% of the time If that seems surprising, it is worth doing a course like Karpathy's zero to hero NN, and have all the magic peeled away a layer at a time. The reason you can do this is because LLMs don't just generate the next word or token, it produces a probability distribution over all tokens. A JSON parser can give you a list of next valid tokens. The tokens in each case might be…

If you're choosing the next token based on a list of valid next tokens, a uniform random distribution can always generate valid JSON too!

But that's not what an LLM does.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#169

Post by selcuka »

quickthrower2 wrote: post #153
coder543 wrote: post #145

Earlier quoted context omitted.

> at least from my experience of GPT3.5 (not used 4 much). And 4 is tremendously better than 3.5, in my own experience. Not perfect, but actually useful.

Can anyone recommend a good, and trusted UI so I can use it via the API? I don't want to pay monthly for it, but would be nice to use occasionally. I keep meaning to do this!

OpenAI has its own playground where you can test all models (I believe GPT-4 is not available to everyone yet):

https://platform.openai.com/playground

Monthly subscription is only for ChatGPT. When you use the APIs you pay per token.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#170

Post by fenomas »

OJFord wrote: post #160
quickthrower2 wrote: post #130

> LLMs can generate valid JSON 100% of the time If that seems surprising, it is worth doing a course like Karpathy's zero to hero NN, and have all the magic peeled away a layer at a time. The reason you can do this is because LLMs don't just generate the next word or token, it produces a probability distribution over all tokens. A JSON parser can give you a list of next valid tokens. The tokens in each case might be…

Maybe it's just me, but I'm not doing anything that calls itself 'zero to hero'. Would love some good resources (preferably textbook, or at least written) on LLMs though. I don't even understand the link to 'generative' image/video AI, which seems to have exploded at roughly the same time and surely isn't a coincidence. I studied a little (literally 'intro to') ML at university, about enough to grok it as an applicat…

The zero to hero video series is what you're looking for - look past the name and watch it. It's excellent.
Post reply on HN