Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sampling and structured outputs in LLMs

parthsareen.com

Post reply on HN
Print view View original

91–99 of 99 posts

Re: Sampling and structured outputs in LLMs

#91

Post by dcreater »

FlyingLawnmower wrote: post #69
dcreater wrote: post #50

Earlier quoted context omitted.

How does this compare to pydantic ai? I'm yet to see a thorough comparison of design, performance and reliability between these options (along with outlines etc)

We did quite a thorough benchmarking of various structured decoding providers in one of our papers: https://arxiv.org/abs/2501.10868v3 , measuring structured outputs providers on performance, constraint flexibility, downstream task accuracy, etc. Happy to chat more about the benchmark. Note that these are a bit out of date though, I'm sure many of the providers we tested have made improvements (and some have switched…

thanks for the paper link! Im surprised there is such a minimal improvement in structured outputs when using any of these tools over the bare LLM!

Re: Sampling and structured outputs in LLMs

#92

Post by dcreater »

stillsut wrote: post #73
FlyingLawnmower wrote: post #22

I spent a couple years building a high performance, expressive library for structured outputs in LLMs. Our library is used by OpenAI for structured outputs on the hosted API. Happy to answer questions on how this works: User friendly library that connects to lots of OSS model serving backends: https://github.com/guidance-ai/guidance/ Core Rust library written for high performance mask computation (written mostly by m…

I'm also working on a library to steer the sampling step of LLM's but more for steganographic / arbitrary data encoding purposes. Should work with any llama.cpp compatible model: https://github.com/sutt/innocuous

i am not following how you encoded a BTC address into a poem. can you help explain?

Re: Sampling and structured outputs in LLMs

#93

Post by parthsareen »

frotaur wrote: post #6

When doing structured sampling, why is the token sampled, checked against the grammar, and resampled if it's wrong by applying the mask ? Why wouldn't we apply the mask immediately for the first sampling? Is this an optimization somehow, is masking expensive?

Hey! I'm the author of the post. We haven't optimized sampling yet so it's running linearly on the CPU. A lot of SOTA work either does this while the model is running the forward pass or does the masking on the GPU.

The greedy accept is so that the mask doesn't need to be computed. Planning to make this more efficient from either ends.

Re: Sampling and structured outputs in LLMs

#94

Post by parthsareen »

EnnEmmEss wrote: post #89

This is off-tangent but I find it a bit odd that the blog uses a URL fragment to load different articles when it's usually used to navigate within a page. A consequence of this seems to be that clicking the link to a different article leaves you at the bottom of the page even though the article itself has changed. This seems to be using JS to fetch the markdown and then render it but I do feel that it may be better o…

That's a great idea. Going to try this next :)

Re: Sampling and structured outputs in LLMs

#95

Post by EnnEmmEss »

parthsareen wrote: post #94
EnnEmmEss wrote: post #89

This is off-tangent but I find it a bit odd that the blog uses a URL fragment to load different articles when it's usually used to navigate within a page. A consequence of this seems to be that clicking the link to a different article leaves you at the bottom of the page even though the article itself has changed. This seems to be using JS to fetch the markdown and then render it but I do feel that it may be better o…

That's a great idea. Going to try this next :)

Happy to help :)

Re: Sampling and structured outputs in LLMs

#97

Post by msp26 »

Inviz wrote: post #96
msp26 wrote: post #66

Earlier quoted context omitted.

that only works for the outer level, not for any nested fields

nested field have their own propertyOrdering

can you provide an example please? The docs suggest that propertyOrdering can only be a list[str].

Re: Sampling and structured outputs in LLMs

#98

Post by Inviz »

msp26 wrote: post #97
Inviz wrote: post #96

Earlier quoted context omitted.

nested field have their own propertyOrdering

can you provide an example please? The docs suggest that propertyOrdering can only be a list[str].

yes but you can put it on nested objects. Each type: object can have its own propertyOrdering, so you control whole tree

Re: Sampling and structured outputs in LLMs

#99

Post by stillsut »

dcreater wrote: post #92
stillsut wrote: post #73

Earlier quoted context omitted.

I'm also working on a library to steer the sampling step of LLM's but more for steganographic / arbitrary data encoding purposes. Should work with any llama.cpp compatible model: https://github.com/sutt/innocuous

i am not following how you encoded a BTC address into a poem. can you help explain?

I think the easiest explanation is to look at the table here: https://github.com/sutt/innocuous?tab=readme-ov-file#how-it-...

Watch how the "Cumulative encoding" row grows each iteration (that's where the BTC address will be encoded) and then look at the other rows for how the algorithm arrives at that.

Thanks for checking it out!

Post reply on HN