Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sampling and structured outputs in LLMs

parthsareen.com

Post reply on HN
Print view View original

81–90 of 99 posts

Re: Sampling and structured outputs in LLMs

#81

Post by 0x4FFC8F »

FlyingLawnmower wrote: post #69
dcreater wrote: post #50

Earlier quoted context omitted.

How does this compare to pydantic ai? I'm yet to see a thorough comparison of design, performance and reliability between these options (along with outlines etc)

We did quite a thorough benchmarking of various structured decoding providers in one of our papers: https://arxiv.org/abs/2501.10868v3 , measuring structured outputs providers on performance, constraint flexibility, downstream task accuracy, etc. Happy to chat more about the benchmark. Note that these are a bit out of date though, I'm sure many of the providers we tested have made improvements (and some have switched…

I think @dcreater was asking how these various structee decoding providers compare with how pydantic ai handles structured output, i.e via tool calling, forcing the LLM to use a tool and its arguments are a json schema hence you read the tool call arguments and get a structured output.

Re: Sampling and structured outputs in LLMs

#82

Post by dcreater »

_andrei_ wrote: post #76
dcreater wrote: post #50

Earlier quoted context omitted.

How does this compare to pydantic ai? I'm yet to see a thorough comparison of design, performance and reliability between these options (along with outlines etc)

pydantic is a _validation_ library, it does not do any kind of constraints by itself

im referring to pydanticai https://ai.pydantic.dev/

Re: Sampling and structured outputs in LLMs

#83

Post by snerbles »

barrkel wrote: post #19
thrance wrote: post #5

It's still baffling to me that the various API providers don't let us upload our custom grammars. It would enable so many use cases, like HTML generation for example, at essentially no cost on their part.

Using grammar constrained output in llama.cpp - which has been available for ages and I think is a different implementation to the one described here - does slow down generation quite a bit. I expect it has a naive implementation. As to why providers don't give you a nice API, maybe it's hard to implement efficiently. It's not too bad if inference is happening token by token and reverting to the CPU every time, but I…

A guide on llama.cpp's grammars (nine hours and not a single mention of "GBNF"? HN is slipping) is here:

https://github.com/ggml-org/llama.cpp/blob/master/grammars/R...

There's also a grammar validation tool in the default llama.cpp build, which is much easier to reason about for debugging grammars than having them bounce off the server.

Re: Sampling and structured outputs in LLMs

#84

Post by hansvm »

matheist wrote: post #65
hansvm wrote: post #59

Earlier quoted context omitted.

That's definitely possible. As you know, (most current) LLMs build text autoregressively. This allows them to generate text with _exactly_ the same distribution as the training data. When you constrain LLM output at each token, that gives a completely different distribution from letting the LLM generate a full output and then doing something with that (trying again, returning an error, post-processing, etc). E.g.: Su…

Why do you think the constrained percentages are 0/75/25 and not eg 0/66/33? (ie same relative likelihood for valid outputs)

The constraint algorithm looks something like:

1. Choose the first token. If well-trained you have a 75% chance of choosing "a" and a 25% chance of choosing "b". Both are valid for that grammar.

2. Choose the second token. Regardless of your first token there is exactly once choice of grammar-adhering completion. You're now at a 75% chance of "ab" and a 25% chance of "ba" (mirroring the first-token chance).

For a toy example like this you obviously wouldn't use an LLM, but techniques like you're suggesting don't work because it's infeasible to enumerate all the valid outputs and re-weight and because greedy and semi-greedy strategies aren't anywhere near sufficient to side-step the issue. At the point in time you select the "a" token at a 75% probability it's game-over unless you re-run the LLM. You can't beam search either (doing so just changes which token you'll mis-predict, and even then only for very local grammar mistakes).

Looking at my JSON example from earlier, a beam search to avoid that re-weighting requires a depth of at least 4 (going as far as the ellipsis plus the stop token), and it won't suffice to just consider locally high-weight paths (you can probably hack something together for that one issue in particular which searches high weight paths and backtracks if they're found to be low-weight due to grammar mismatches, but that has its own bias unless you fan out to all 1e19 length-4 paths, and it won't solve the general problem regardless).

Phrased slightly differently, you don't have a compute_future_grammar_adhering_weight(token) function which is tractably computable, so you can't actually redistribute the 8.3% probability from the "a" branch to the "b" branch.

Re: Sampling and structured outputs in LLMs

#85

Post by matheist »

hansvm wrote: post #84
matheist wrote: post #65

Earlier quoted context omitted.

Why do you think the constrained percentages are 0/75/25 and not eg 0/66/33? (ie same relative likelihood for valid outputs)

The constraint algorithm looks something like: 1. Choose the first token. If well-trained you have a 75% chance of choosing "a" and a 25% chance of choosing "b". Both are valid for that grammar. 2. Choose the second token. Regardless of your first token there is exactly once choice of grammar-adhering completion. You're now at a 75% chance of "ab" and a 25% chance of "ba" (mirroring the first-token chance). For a toy…

Oh now I understand. I thought your ab and ba were single tokens (even though that doesn't make sense in context). Once you point out they're separate tokens, I follow you. Thank you!

Edit: that's a great example

Edit 2: even more fun: training data is [ab, ab, ba, bb, bb, bb]. Then constrained sampling flips your likelihood from 1:2 to 2:1

Re: Sampling and structured outputs in LLMs

#86

Post by ijk »

FlyingLawnmower wrote: post #71
ijk wrote: post #37

Earlier quoted context omitted.

I've been curious about grammar support for non-JSON applications. (i.e., I have some use cases where XML is more natural and easier to parse but Pydantic seems to assume you should only work with JSON.) Would guidance be able to handle this use case? In general I find that matching the most natural format for a document outperforms waiting for the big model trainers to convince the model that the format you want is…

guidance can handle many context-free grammars. We use an Earley parser under the hood ( https://en.wikipedia.org/wiki/Earley_parser ) which gives us significant flexibility boosts over alternative approaches that use weaker parsers (and went through lots of effort to make Earley parsing fast enough to not slow down LM inference). However, XML is not perfectly context-free, though with some basic assumptions you can…

I'm really just looking for a subset of XML so that's probably sufficient.

For me, the advantage that Pydantic AI has right now is that it's easy to do ingestion/validation of the generated text, since I've already got the typing information in place. If I had similar ways to create new specialized grammars on the fly (e.g., I want XML-ish tags with these fields, but also allow for arbitrary additional fields...) that would significantly sway my implementation decisions.

Re: Sampling and structured outputs in LLMs

#87

Post by lubujackson »

miki123211 wrote: post #54
madethemcry wrote: post #47

That was as great reading, thank you. I've a related observation. In my experience the amount of hallucinated urls with structured output (think of a field `url` or `link`) is pretty high. Especially compared to the alternative approach, where you let the llm generate text and then use a second llm to convert the text into the desired structured format. With structured output, it's like the llm is forced to answer in…

What I've found is that it is very important to make structured outputs as easy for the LLM as possible. This means making your schemas LLM-friendly instead of programmer-friendly. E.g. if the LLM hallucinates non-existing URLs, you may add a boolean "contains_url" field to your entity's JSON schema, placing it before the URL field itself . This way, the URL extraction is split into two simpler steps, checking if the…

Awesome insight, thanks for this!

Re: Sampling and structured outputs in LLMs

#88

Post by Balgair »

FlyingLawnmower wrote: post #22

I spent a couple years building a high performance, expressive library for structured outputs in LLMs. Our library is used by OpenAI for structured outputs on the hosted API. Happy to answer questions on how this works: User friendly library that connects to lots of OSS model serving backends: https://github.com/guidance-ai/guidance/ Core Rust library written for high performance mask computation (written mostly by m…

I'm stupid, so my question will be too.

I'm trying to write a really large book. I have a lot of material that I'm using RAG to help manage. I put into my prompts the top RAG cosine scores with some summaries of characters and previous chapters and scene sketches. I get scenes out and then work them over. LLMs are really helpful for my disability and have allowed me to make any progress at all on this.

Is your thing something I should look into for helping keep track of my material. I'm using Excel sheets and crappy python code right now.

Im pretty sure your stuff is some super technical backend thingy, but I figured I'd shoot my shot here. Thanks for any and all info, I appreciate it

Re: Sampling and structured outputs in LLMs

#89

Post by EnnEmmEss »

This is off-tangent but I find it a bit odd that the blog uses a URL fragment to load different articles when it's usually used to navigate within a page.

A consequence of this seems to be that clicking the link to a different article leaves you at the bottom of the page even though the article itself has changed.

This seems to be using JS to fetch the markdown and then render it but I do feel that it may be better off to simply pre-convert the markdown as part of the deployment process and serve the static page.

Re: Sampling and structured outputs in LLMs

#90

Post by hansvm »

matheist wrote: post #85
hansvm wrote: post #84

Earlier quoted context omitted.

The constraint algorithm looks something like: 1. Choose the first token. If well-trained you have a 75% chance of choosing "a" and a 25% chance of choosing "b". Both are valid for that grammar. 2. Choose the second token. Regardless of your first token there is exactly once choice of grammar-adhering completion. You're now at a 75% chance of "ab" and a 25% chance of "ba" (mirroring the first-token chance). For a toy…

Oh now I understand. I thought your ab and ba were single tokens (even though that doesn't make sense in context). Once you point out they're separate tokens, I follow you. Thank you! Edit: that's a great example Edit 2: even more fun: training data is [ab, ab, ba, bb, bb, bb]. Then constrained sampling flips your likelihood from 1:2 to 2:1

Thanks :) My example is minimal, which is a little nice since I wind up re-deriving it in a hurry every time I need it. I do like the 1:2 to 2:1 symmetry though. Very elegant.
Post reply on HN