Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: LLMs can generate valid JSON 100% of the time

github.com

Post reply on HN
Print view View original

271–280 of 315 posts

Re: Show HN: LLMs can generate valid JSON 100% of the time

#271

Post by IanCal »

newhouseb wrote: post #269
druskacik wrote: post #237

Earlier quoted context omitted.

In this case (multiple choice generation), if one of the possible outputs does no match the regex, you can just exclude it from generation. I am trying to think of an example where "answer prefix might have been extremely unlikely to yield a valid response, but the technique ( ... ) constructs a valid response from it regardless" , which might really cause a problem. But to no luck. Anyone has any idea? This could po…

An example from an earlier comment of mine on a different thread (assuming I've understood correctly): > let's say we had a grammar that had a key "healthy" with values "very_unhealthy" or "moderately_healthy." For broccoli, the LLM might intend to say "very_healthy" and choose "very" but then be pigeonholed into saying "very_unhealthy" because it's the only valid completion according to the grammar. That said, you c…

This is a concern of mine, as well as limiting the amount that an LLM can talk through a problem - sometimes to nothing. Getting them to work through things IMO dramatically improves their output.

My gut feeling is that taking the output and if it's broken then start fixing it would have a better result - you could even then completely limit the output to only valid json. For your example, if it wrote "very_healthy" and was given an error message explaining that this wasn't an option it had to choose from very_unhealthy" or "moderately_healthy" I would expect a halfway decent model to pick "moderately_healthy".

This has the benefit of allowing you to use a more powerful model for reasoning (like GPT4) and a local model where you can do this kind of token probability manipulation for just fixing the data.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#272

Post by tomduncalf »

ipaddr wrote: post #207
selcuka wrote: post #172

Earlier quoted context omitted.

The premise of function calling is great, but in my experience (at least on GPT-3.5, haven't tried it with GPT-4 yet) it seems to generate wildly different, and less useful results, for the same prompt.

You can change the randomness value to 0 and get the same output each time for the same text

In my experience (with GPT-4 at least), a temperature of 0 does not result in deterministic output. It's more consistent but outputs do still vary for the same input. I feel like temperature is a bit more like "how creative should the model be?"

Re: Show HN: LLMs can generate valid JSON 100% of the time

#276

Post by vlovich123 »

lexandstuff wrote: post #259
vlovich123 wrote: post #215

How is this different from generating such things without an LLM? In other words picking random valid tokens from the grammar via fuzzing or similar techniques.

LLMS allows for building systems that take user requests in text: "book the next flight to Egpyt" and convert them into a system message: `{"action": "book_flight", "destination": "Egypt", ... }` However, anyone who's tried to build a system like this on GPT or other LLM soon learns that they don't always do as they're told, and it can be hard to get them to return valid JSON or correct instructions translation relia…

Why bother with conversion to JSON directly from the LLM instead of a simpler format (eg line separated) that you then convert into JSON normally?

Re: Show HN: LLMs can generate valid JSON 100% of the time

#277

Post by remilouf »

lettergram wrote: post #87

Few thoughts, you're effectively creating representations that can convert to JSON (kudos!) Can't mention how we did it (there are a lot of public patents, if interested), but back in 2018 we had a way to generate synthetic data (statistically, structurally similar) off any dataset - https://medium.com/capital-one-tech/why-you-dont-necessarily... You could also design datasets if you wanted. It'd keep similar relatio…

Thank you for the pointer. The best part of posting on HN is the long list of related work you get in response.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#278

Post by remilouf »

spott wrote: post #55
remilouf wrote: post #10

Earlier quoted context omitted.

Two differences: (1) This feature only requires regex-guided generation. We have a PR for BNF sampling that is about to be merged. (2) ggml loops over the entire vocabulary (~50k tokens) at each step, which introduces a noticeable overhead, and makes it unusable for complex grammars. Our method works by building an index at initialization, and build the masks at each step with a dictionary lookup. Once the index is b…

Regex-guided gen is slick… is it arbitrary? Or are you custom building it for json? If arbitrary, how are you pre-defining a set of masks? I would expect that splitting an arbitrary regex into a bunch of contexts for a masking dictionary to be non-trivial.

Regex-Gen is implemented in all generality in the library (minus some constructs that we still have to add). JSON is merely an application.

You can read https://blog.normalcomputing.ai/posts/2023-07-27-regex-guide... for a more detailed explanation of how it works. Should answer your question :)

Re: Show HN: LLMs can generate valid JSON 100% of the time

#279

Post by remilouf »

burke wrote: post #100
remilouf wrote: post #28

Earlier quoted context omitted.

Thanks for bringing clownfish and relm to my attention! afaik other libraries loop over the entire vocabulary at every step of the generation. We on the other hand build an index at initialization by looping once over the vocabulary. Then generation is just as fast as standard generation.

torch-grammar generates a mask per PDA stack... we don't try to compute all the possible stacks. I'm sure there's something smarter that could be done here and you've probably figured it out (though IIRC regular languages don't have the arbitrarily recursive stack problem that you get when you get to context-free languages?) anyway, in practice we spend a few milliseconds on the first few requests building caches and…

Sorry for misrepresenting your work. Thank you for correcting me and the explanation. Will take a closer look.

Re: Show HN: LLMs can generate valid JSON 100% of the time

#280

Post by YeGoblynQueenne »

IanCal wrote: post #247
YeGoblynQueenne wrote: post #244

Hi, remilouf. You say that your background is in "probabilistic, relational and symbolic programming". In that case I suspect you understand that it is no problem to generate text from a regular or context-free grammar, or really any level of grammar. For example, you can do that very easily in Prolog (a relational language) given a grammar in Definite Clause Grammars notation. As far as I can tell your approach requ…

Wouldn't that generate an entirely random but valid output? Here you want a valid output related to the request. > And why would you need an LLM, a model of natural language, if all you want is to generate structured text, anyway? So that you can parse unstructured text from a person and return structured data for a machine.

>> Wouldn't that generate an entirely random but valid output?

No. Grammars don't generate entirely random output. Even Probabilistic Context Free Grammars can generate deterministic output, depending on how they are sampled. The output can be related to some input, if desired, for example one can give a string with "holes" (variables) as input and have the holes filled-in by the grammar.

>> So that you can parse unstructured text from a person and return structured data for a machine.

If you are willing to spend the effort to write a grammar, you can do that without an LLM.

Post reply on HN