Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sampling and structured outputs in LLMs

parthsareen.com

Post reply on HN
Print view View original

71–80 of 99 posts

Re: Sampling and structured outputs in LLMs

#71

Post by FlyingLawnmower »

ijk wrote: post #37
FlyingLawnmower wrote: post #22

I spent a couple years building a high performance, expressive library for structured outputs in LLMs. Our library is used by OpenAI for structured outputs on the hosted API. Happy to answer questions on how this works: User friendly library that connects to lots of OSS model serving backends: https://github.com/guidance-ai/guidance/ Core Rust library written for high performance mask computation (written mostly by m…

I've been curious about grammar support for non-JSON applications. (i.e., I have some use cases where XML is more natural and easier to parse but Pydantic seems to assume you should only work with JSON.) Would guidance be able to handle this use case? In general I find that matching the most natural format for a document outperforms waiting for the big model trainers to convince the model that the format you want is…

guidance can handle many context-free grammars. We use an Earley parser under the hood (https://en.wikipedia.org/wiki/Earley_parser) which gives us significant flexibility boosts over alternative approaches that use weaker parsers (and went through lots of effort to make Earley parsing fast enough to not slow down LM inference). However, XML is not perfectly context-free, though with some basic assumptions you can make it CF.

The annoying bit with grammars is that they are unfortunately a bit complex to write properly. Fortunately language models are getting better at this, so hopefully to get an XML grammar, you can get most of the way there with just a GPT-5 prompt. Suppose it would be a good idea to have a better pre-built set of popular grammars (like a modified XML) in guidance so that we cut this headache out for users...!

Re: Sampling and structured outputs in LLMs

#72

Post by FlyingLawnmower »

ru552 wrote: post #68
FlyingLawnmower wrote: post #22

I spent a couple years building a high performance, expressive library for structured outputs in LLMs. Our library is used by OpenAI for structured outputs on the hosted API. Happy to answer questions on how this works: User friendly library that connects to lots of OSS model serving backends: https://github.com/guidance-ai/guidance/ Core Rust library written for high performance mask computation (written mostly by m…

"The constraint system offered by Guidance is extremely powerful. It can ensure that the output conforms to any context free grammar (so long as the backend LLM has full support for Guidance). More on this below." --from https://github.com/guidance-ai/guidance/ I didn't find any more on that comment below. Is there a list of supported LLMs?

Good point re: documentation...

We have support for Huggingface Transformers, llama.cpp, vLLM, SGLang, and TensorRT-LLM, along with some smaller providers (e.g. mistral.rs). Using any of these libraries as an inference host means you can use an OSS model with the guidance backend for full support. Most open source models will run on at least one of these backends (with vLLM probably being the most popular hosted solution, and transformers/llama.cpp being the most popular local model solutions)

We're also the backend used by OpenAI/Azure OpenAI for structured outputs on the closed source model side.

Re: Sampling and structured outputs in LLMs

#73

Post by stillsut »

FlyingLawnmower wrote: post #22

I spent a couple years building a high performance, expressive library for structured outputs in LLMs. Our library is used by OpenAI for structured outputs on the hosted API. Happy to answer questions on how this works: User friendly library that connects to lots of OSS model serving backends: https://github.com/guidance-ai/guidance/ Core Rust library written for high performance mask computation (written mostly by m…

I'm also working on a library to steer the sampling step of LLM's but more for steganographic / arbitrary data encoding purposes.

Should work with any llama.cpp compatible model: https://github.com/sutt/innocuous

Re: Sampling and structured outputs in LLMs

#74

Post by heckintime »

I've found that writing a very simple DSL that resembles human speech and an interpreter that can output JSON is very effective.

Human

4x1200 with 30 second rest

AI DSL output

Repeat 4 times:

- Run 1200 meters

- Rest 30 seconds

I hand wrote a recursive descent parser in Python to process DSL. Human speech to DSL is pretty effective with a simple prompt and some examples.

I created a tool that can program Garmin & Apple Watches for interval training based on what I wrote above.

https://speedystride.com

Looking for beta testers- please give it a try :)

Re: Sampling and structured outputs in LLMs

#76

Post by _andrei_ »

dcreater wrote: post #50
FlyingLawnmower wrote: post #22

I spent a couple years building a high performance, expressive library for structured outputs in LLMs. Our library is used by OpenAI for structured outputs on the hosted API. Happy to answer questions on how this works: User friendly library that connects to lots of OSS model serving backends: https://github.com/guidance-ai/guidance/ Core Rust library written for high performance mask computation (written mostly by m…

How does this compare to pydantic ai? I'm yet to see a thorough comparison of design, performance and reliability between these options (along with outlines etc)

pydantic is a _validation_ library, it does not do any kind of constraints by itself

Re: Sampling and structured outputs in LLMs

#78

Post by michaelgiba »

viralpraxis wrote: post #55
NitpickLawyer wrote: post #36

Earlier quoted context omitted.

Perhaps I worded that poorly. What I mean by semantic correctness is that the model could output nonsensical values for some things. Say in a game, "normal" health is ~100hp and the model creates a wizard with 50hp but then a mouse with 10000hp. So you're guaranteed to get a parsable json object (syntactically correct) but what the values are in that json is not guaranteed to make sense in the given context.

You can specify `minimum` and `maximum` property for these fields. So this schema { "$id": "https://example.com/test.schema.json", "$schema": "https://json-schema.org/draft/2020-12/schema", "title": "Person", "type": "object", "properties": { "hp": { "type": "integer", "description": "HP", "minimum": 1, "maximum": 15 } } } is converted to this BNF-like representation: hp ::= ([1-9] | "1" [0-5]) space hp-kv ::= "\"hp\…

For anyone curious here is an interactive write up about this http://michaelgiba.com/grammar-based/index.html

Re: Sampling and structured outputs in LLMs

#80

Post by parthsareen »

iandanforth wrote: post #21

This is a great writeup! There was a period where reliable structured output was a significant differentiator and was the 'secret sauce' behind some companies success. A NL->SQL company I am familiar with comes to mind. Nice to see this both public and supported by a growing ecosystem of libraries. One statement surprised me was that the author thinks "models over time will just be able to output JSON perfectly witho…

Thank you! Maybe not "perfect" but near-perfect is something we can expect. Models like the Osmosis structure which just structure data inspired some of that thinking (https://ollama.com/Osmosis/Osmosis-Structure-0.6B). Historically, JSON generation has been a latent capability of a model rather than a trained one, but that seems to be changing. gpt-oss was particularly trained for this type of behavior and so the token probabilities are heavily skewed to conform to JSON. Will be interesting to see the next batch of models!
Post reply on HN