Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sampling and structured outputs in LLMs

parthsareen.com

Post reply on HN
Print view View original

61–70 of 99 posts

Re: Sampling and structured outputs in LLMs

#61

Post by visarga »

I was hoping to find some insights about why performance drops when using actual structured outputs. It's been a known problem. For example this paper "Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models" says:

> Surprisingly, we observe a significant decline in LLMs’ reasoning abilities under format restrictions. Furthermore, we find that stricter format constraints generally lead to greater performance degradation in reasoning tasks.

https://arxiv.org/abs/2408.02442v1

Re: Sampling and structured outputs in LLMs

#62

Post by 2THFairy »

nl wrote: post #38
2THFairy wrote: post #26

Earlier quoted context omitted.

There are some implementation concerns, but the real answer is that it is an ideological choice. The AI companies believe that these kinds of grammar mistakes will be solved by improving the models. To build out tools for grammar constrained inference like this is to suggest, on some level, that GPT-N+1 won't magically solve the problem. The deeper level is that it's not just simple grammar constraints. Constraining…

Most API providers (Together, Fireworks etc) don't build their own models.

You don't need a new model. The trick of the technique is that you only change how tokens are sampled; Zero out the probability of every token that would be illegal under the grammar or other constraints.

All you need for that is an inference API that gives you the full output vector, which is trivial for any model you run on your own hardware.

Re: Sampling and structured outputs in LLMs

#63

Post by anentropic »

madethemcry wrote: post #47

That was as great reading, thank you. I've a related observation. In my experience the amount of hallucinated urls with structured output (think of a field `url` or `link`) is pretty high. Especially compared to the alternative approach, where you let the llm generate text and then use a second llm to convert the text into the desired structured format. With structured output, it's like the llm is forced to answer in…

> let the llm generate text and then use a second llm to convert the text into the desired structured format

this sounds similar to what they discussed in the article with regards to "thinking" models, i.e. let them generate their blah blah preamble first before starting to constrain the output to structured format

Re: Sampling and structured outputs in LLMs

#64

Post by roadside_picnic »

visarga wrote: post #61

I was hoping to find some insights about why performance drops when using actual structured outputs. It's been a known problem. For example this paper "Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models" says: > Surprisingly, we observe a significant decline in LLMs’ reasoning abilities under format restrictions. Furthermore, we find that stricter format constrai…

That paper had some serious methodological issues and the results have been shown to be misunderstood/incorrect in the majority of cases. In fact, in many cases structured outputs have shown to improve the quality of the results from an LLM (at least in terms of evaluation performance). The team at behind the Outlines library released a response the covers the issues in details and provides more information about structured outputs [0].

0. https://blog.dottxt.ai/say-what-you-mean.html

Re: Sampling and structured outputs in LLMs

#65

Post by matheist »

hansvm wrote: post #59
madethemcry wrote: post #47

That was as great reading, thank you. I've a related observation. In my experience the amount of hallucinated urls with structured output (think of a field `url` or `link`) is pretty high. Especially compared to the alternative approach, where you let the llm generate text and then use a second llm to convert the text into the desired structured format. With structured output, it's like the llm is forced to answer in…

That's definitely possible. As you know, (most current) LLMs build text autoregressively. This allows them to generate text with _exactly_ the same distribution as the training data. When you constrain LLM output at each token, that gives a completely different distribution from letting the LLM generate a full output and then doing something with that (trying again, returning an error, post-processing, etc). E.g.: Su…

Why do you think the constrained percentages are 0/75/25 and not eg 0/66/33? (ie same relative likelihood for valid outputs)

Re: Sampling and structured outputs in LLMs

#66

Post by msp26 »

Inviz wrote: post #41
minimaxir wrote: post #32

Google's Gemini API is a bit odd with structured outputs. If you specify an Application/JSON response mimetype, it will reliably respond with a consistent JSON output without any prompt engineering shenanigans. For my workflows, this setting plus providing a JSON Schema in the system prompt works even with complex schema. The Gemini API has a canonical implementation of structured outputs where you can instead pass t…

gemini api has propertyOrdering field for that

that only works for the outer level, not for any nested fields

Re: Sampling and structured outputs in LLMs

#67

Post by FlyingLawnmower »

ninadpathak wrote: post #28
FlyingLawnmower wrote: post #22

I spent a couple years building a high performance, expressive library for structured outputs in LLMs. Our library is used by OpenAI for structured outputs on the hosted API. Happy to answer questions on how this works: User friendly library that connects to lots of OSS model serving backends: https://github.com/guidance-ai/guidance/ Core Rust library written for high performance mask computation (written mostly by m…

[dead]

Thanks :)

Great question re: adoption...it's definitely dominated by JSON. Most API providers have standardized on JSON outputs, so application teams have started building shims that map other formats to JSON and back. Similarly, with models heavily being post-trained to generate "good" JSON, I think there's a better model-constraint alignment story with JSON than most arbitrary grammars.

That said, internally, we experiment quite a lot with custom grammars all across the stack. It's more complicated to write a grammar than a JSON schema (though LMs are very good at grammar writing now) and more error prone to debug, but it can help significantly in certain cases (e.g. having models write custom DSLs not commonly found on the internet, at various parts of a model training pipeline, etc. etc.). I'm hoping that with the right tooling around it, the broader community will start nudging beyond JSON.

To that end, the python guidance library is really an attempt to make writing grammars more friendly to a python programmer. More to be done here of course!

Re: Sampling and structured outputs in LLMs

#68

Post by ru552 »

FlyingLawnmower wrote: post #22

I spent a couple years building a high performance, expressive library for structured outputs in LLMs. Our library is used by OpenAI for structured outputs on the hosted API. Happy to answer questions on how this works: User friendly library that connects to lots of OSS model serving backends: https://github.com/guidance-ai/guidance/ Core Rust library written for high performance mask computation (written mostly by m…

"The constraint system offered by Guidance is extremely powerful. It can ensure that the output conforms to any context free grammar (so long as the backend LLM has full support for Guidance). More on this below." --from https://github.com/guidance-ai/guidance/

I didn't find any more on that comment below. Is there a list of supported LLMs?

Re: Sampling and structured outputs in LLMs

#69

Post by FlyingLawnmower »

dcreater wrote: post #50
FlyingLawnmower wrote: post #22

I spent a couple years building a high performance, expressive library for structured outputs in LLMs. Our library is used by OpenAI for structured outputs on the hosted API. Happy to answer questions on how this works: User friendly library that connects to lots of OSS model serving backends: https://github.com/guidance-ai/guidance/ Core Rust library written for high performance mask computation (written mostly by m…

How does this compare to pydantic ai? I'm yet to see a thorough comparison of design, performance and reliability between these options (along with outlines etc)

We did quite a thorough benchmarking of various structured decoding providers in one of our papers: https://arxiv.org/abs/2501.10868v3 , measuring structured outputs providers on performance, constraint flexibility, downstream task accuracy, etc.

Happy to chat more about the benchmark. Note that these are a bit out of date though, I'm sure many of the providers we tested have made improvements (and some have switched to wholesale using llguidance as a backend)

Re: Sampling and structured outputs in LLMs

#70

Post by lelanthran »

FlyingLawnmower wrote: post #22

I spent a couple years building a high performance, expressive library for structured outputs in LLMs. Our library is used by OpenAI for structured outputs on the hosted API. Happy to answer questions on how this works: User friendly library that connects to lots of OSS model serving backends: https://github.com/guidance-ai/guidance/ Core Rust library written for high performance mask computation (written mostly by m…

> Happy to answer questions on how this works

Well, thank you for that; from a quick skim of Guidance, it looks like it is used when interfacing with the model directly - i.e. if I want to use Guidance I can't simply send input to my local Ollama instance, I have to stand up a small Python program that loads the model, accepts input from the user, push the user input tokens into the model, and for each output token, reject it if it fails some criteria.

Is this correct? If so, it means that the current way LLMs are interfaced with (via stdin/stout or an HTTP endpoint) can't be used with something like Guidance, correct?

Post reply on HN