Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sampling and structured outputs in LLMs

parthsareen.com

Post reply on HN
Print view View original

21–30 of 99 posts

Re: Sampling and structured outputs in LLMs

#21

Post by iandanforth »

This is a great writeup! There was a period where reliable structured output was a significant differentiator and was the 'secret sauce' behind some companies success. A NL->SQL company I am familiar with comes to mind. Nice to see this both public and supported by a growing ecosystem of libraries.

One statement surprised me was that the author thinks "models over time will just be able to output JSON perfectly without the need for constraining over time."

I'm not sure how this conclusion was reached. "Perfectly" is a bar that probabilistic sampling cannot meet.

Re: Sampling and structured outputs in LLMs

#22

Post by FlyingLawnmower »

I spent a couple years building a high performance, expressive library for structured outputs in LLMs. Our library is used by OpenAI for structured outputs on the hosted API. Happy to answer questions on how this works:

User friendly library that connects to lots of OSS model serving backends: https://github.com/guidance-ai/guidance/

Core Rust library written for high performance mask computation (written mostly by my collaborator @mmoskal): http://github.com/guidance-ai/llguidance

Re: Sampling and structured outputs in LLMs

#23

Post by FlyingLawnmower »

electroglyph wrote: post #12

If the current position in the structure only has one possibility (like a comma, bracket, etc.) do you just force that as the next token and continue?

We do enable forcing these sequences of tokens in guidance, and find that it significantly speeds up structured generation. There are tricky alignment issues to make sure you pick the right sequence of tokens, but you can often proxy this well by using the model's native tokenizer. Some details here in an old blog: https://guidance.readthedocs.io/en/latest/example_notebooks/...

Re: Sampling and structured outputs in LLMs

#24

Post by FlyingLawnmower »

thrance wrote: post #5

It's still baffling to me that the various API providers don't let us upload our custom grammars. It would enable so many use cases, like HTML generation for example, at essentially no cost on their part.

OpenAI has started to (at least for tool calls): https://platform.openai.com/docs/guides/function-calling#con...

Re: Sampling and structured outputs in LLMs

#25

Post by FlyingLawnmower »

barrkel wrote: post #19
thrance wrote: post #5

It's still baffling to me that the various API providers don't let us upload our custom grammars. It would enable so many use cases, like HTML generation for example, at essentially no cost on their part.

Using grammar constrained output in llama.cpp - which has been available for ages and I think is a different implementation to the one described here - does slow down generation quite a bit. I expect it has a naive implementation. As to why providers don't give you a nice API, maybe it's hard to implement efficiently. It's not too bad if inference is happening token by token and reverting to the CPU every time, but I…

If your masking is fast enough, you can make it easily work with spec dec too :). We manage to keep this on CPU. Some details here: https://github.com/guidance-ai/llguidance/blob/main/docs/opt...

Re: Sampling and structured outputs in LLMs

#26

Post by 2THFairy »

thrance wrote: post #5

It's still baffling to me that the various API providers don't let us upload our custom grammars. It would enable so many use cases, like HTML generation for example, at essentially no cost on their part.

There are some implementation concerns, but the real answer is that it is an ideological choice.

The AI companies believe that these kinds of grammar mistakes will be solved by improving the models. To build out tools for grammar constrained inference like this is to suggest, on some level, that GPT-N+1 won't magically solve the problem.

The deeper level is that it's not just simple grammar constraints. Constraining to JSON is a nice party trick, but it opens the door to further ideas. How about constraining to a programming language's grammar? Those are well defined, you just swap the JSON grammar file for the Java grammar file, job done.

We can go further: Why not use a language server to constrain not only the grammar but also the content? What variables and functions are in-scope is known, constraining a variable reference or function call to one of their names can be done with the same techique as grammar constraints. ("monitor-guided decoding", figured out back in 2023)

Entire classes of hallucination problems can be eliminated this way. The marketing writes itself; "Our AI is literally incapable of making the errors humans make!"

What many AI developers, firms, and especially their leaders find grating about this is the implication. That AI is fallible and has to be constrained.

Another such inconvenience is that while these techniques improve grammar they highlight semantic problems. The code is correct & compiles, it just does the wrong thing.

Re: Sampling and structured outputs in LLMs

#27

Post by FlyingLawnmower »

frotaur wrote: post #6

When doing structured sampling, why is the token sampled, checked against the grammar, and resampled if it's wrong by applying the mask ? Why wouldn't we apply the mask immediately for the first sampling? Is this an optimization somehow, is masking expensive?

If you can screen tokens against your grammar fast enough, you can build a bitmask over the entire token vocabulary and apply it right before sampling. As vocabulary sizes grow, this gets more complex to do in real time, but we (and other libraries) have found several optimizations to do this extremely quickly (eg for guidance, we detail some optimizations here https://github.com/guidance-ai/llguidance/blob/main/docs/opt...).

Other libraries work by essentially pre-computing all the masks for all possible generations, but of course you're restricted to working with simple grammars in this case (like a subset of regular expressions)

Re: Sampling and structured outputs in LLMs

#28

Post by ninadpathak »

FlyingLawnmower wrote: post #22

I spent a couple years building a high performance, expressive library for structured outputs in LLMs. Our library is used by OpenAI for structured outputs on the hosted API. Happy to answer questions on how this works: User friendly library that connects to lots of OSS model serving backends: https://github.com/guidance-ai/guidance/ Core Rust library written for high performance mask computation (written mostly by m…

[dead]

Re: Sampling and structured outputs in LLMs

#29

Post by ninadpathak »

iandanforth wrote: post #21

This is a great writeup! There was a period where reliable structured output was a significant differentiator and was the 'secret sauce' behind some companies success. A NL->SQL company I am familiar with comes to mind. Nice to see this both public and supported by a growing ecosystem of libraries. One statement surprised me was that the author thinks "models over time will just be able to output JSON perfectly witho…

[dead]

Re: Sampling and structured outputs in LLMs

#30

Post by ninadpathak »

frotaur wrote: post #6

When doing structured sampling, why is the token sampled, checked against the grammar, and resampled if it's wrong by applying the mask ? Why wouldn't we apply the mask immediately for the first sampling? Is this an optimization somehow, is masking expensive?

[dead]
Post reply on HN