If the current position in the structure only has one possibility (like a comma, bracket, etc.) do you just force that as the next token and continue?
[dead]
Sampling and structured outputs in LLMs
31–40 of 99 posts
Re: Sampling and structured outputs in LLMs
#32Google's Gemini API is a bit odd with structured outputs. If you specify an Application/JSON response mimetype, it will reliably respond with a consistent JSON output without any prompt engineering shenanigans. For my workflows, this setting plus providing a JSON Schema in the system prompt works even with complex schema.
The Gemini API has a canonical implementation of structured outputs where you can instead pass the JSON schema as a separate parameter to control the grammar more closely. However, this setting will reorder the JSON schema fields to be alphabetical beforehand, which is especially not desired behavior as the order of JSON fields in a schema is often very deliberate to control generation.
Re: Sampling and structured outputs in LLMs
#33Hmm, so if structured output affects the quality of the response maybe it's better to convert the output to a structured format as a post-processing step?
[dead]
Re: Sampling and structured outputs in LLMs
#34It's still baffling to me that the various API providers don't let us upload our custom grammars. It would enable so many use cases, like HTML generation for example, at essentially no cost on their part.
Fireworks does. It is frustrating that AWS/Google/Azure do not.
https://fireworks.ai/docs/structured-responses/structured-ou...
Re: Sampling and structured outputs in LLMs
#35I spent a couple years building a high performance, expressive library for structured outputs in LLMs. Our library is used by OpenAI for structured outputs on the hosted API. Happy to answer questions on how this works: User friendly library that connects to lots of OSS model serving backends: https://github.com/guidance-ai/guidance/ Core Rust library written for high performance mask computation (written mostly by m…
The LLGuidance paper is highly recommended reading for everyone interested in this! https://guidance-ai.github.io/llguidance/llg-go-brrr
TL;DR instead of just getting a token and seeing if it would be accepted by the parser, you can actually zero-out probabilities for all invalid tokens, and do the computation for this in parallel at effectively zero cost:
> Here, compute_mask() can run on the CPU during the time it would be normally just waiting for the GPU to finish. The line prob[~mask] = 0.0 would normally be fused into the softmax kernel in the last stage of the LLM, with negligible overhead. Therefore, as long as the compute_mask() function completes faster than the LLM forward pass and parser.consume() is negligible (typically follows from compute_mask() speed), the constrained generation will be as fast as the unconstrained one.
I'm curious - have there been any research/conversations about pushing masking even earlier in the pipeline? In theory, there's a fair amount of compute that goes into computing the probability of tokens that will end up being masked away anyways.
Re: Sampling and structured outputs in LLMs
#36Earlier quoted context omitted.
Constrained generation guarantees syntax. It does not guarantee semantic correctness tho. Imagine you want a json object with "hp" and "damage". If you use a grammar, the model will be forced to output a json object with those two values. But it's not guaranteed to get sensible values. With a 2nd pass you basically "condition" it on the text right above, hoping to get better semantic understanding.
I'm pretty sure the grammar is generated from the Json schema, it doesn't just constrain json syntax, it constraints on the schema (including enums and such). The schema is also given to the model (at least in openai) you can put instructions in the json schema as well that will be taken into account.
Perhaps I worded that poorly. What I mean by semantic correctness is that the model could output nonsensical values for some things. Say in a game, "normal" health is ~100hp and the model creates a wizard with 50hp but then a mouse with 10000hp. So you're guaranteed to get a parsable json object (syntactically correct) but what the values are in that json is not guaranteed to make sense in the given context.
Re: Sampling and structured outputs in LLMs
#37I spent a couple years building a high performance, expressive library for structured outputs in LLMs. Our library is used by OpenAI for structured outputs on the hosted API. Happy to answer questions on how this works: User friendly library that connects to lots of OSS model serving backends: https://github.com/guidance-ai/guidance/ Core Rust library written for high performance mask computation (written mostly by m…
I've been curious about grammar support for non-JSON applications. (i.e., I have some use cases where XML is more natural and easier to parse but Pydantic seems to assume you should only work with JSON.) Would guidance be able to handle this use case?
In general I find that matching the most natural format for a document outperforms waiting for the big model trainers to convince the model that the format you want is a valid structure, so anything that lets me interweave structured and unstructured generation is very interesting to me right now.
Re: Sampling and structured outputs in LLMs
#38It's still baffling to me that the various API providers don't let us upload our custom grammars. It would enable so many use cases, like HTML generation for example, at essentially no cost on their part.
There are some implementation concerns, but the real answer is that it is an ideological choice. The AI companies believe that these kinds of grammar mistakes will be solved by improving the models. To build out tools for grammar constrained inference like this is to suggest, on some level, that GPT-N+1 won't magically solve the problem. The deeper level is that it's not just simple grammar constraints. Constraining…
Most API providers (Together, Fireworks etc) don't build their own models.
Re: Sampling and structured outputs in LLMs
#39It's still baffling to me that the various API providers don't let us upload our custom grammars. It would enable so many use cases, like HTML generation for example, at essentially no cost on their part.
There are some implementation concerns, but the real answer is that it is an ideological choice. The AI companies believe that these kinds of grammar mistakes will be solved by improving the models. To build out tools for grammar constrained inference like this is to suggest, on some level, that GPT-N+1 won't magically solve the problem. The deeper level is that it's not just simple grammar constraints. Constraining…
One pattern that I've seen develop (in PydanticAI and elsewhere) is to constrain the output but include an escape hatch. If an error happens, that lets it bail out and report the problem rather than be forced to proceed down a doomed path.
Re: Sampling and structured outputs in LLMs
#40Earlier quoted context omitted.
There are some implementation concerns, but the real answer is that it is an ideological choice. The AI companies believe that these kinds of grammar mistakes will be solved by improving the models. To build out tools for grammar constrained inference like this is to suggest, on some level, that GPT-N+1 won't magically solve the problem. The deeper level is that it's not just simple grammar constraints. Constraining…
Most API providers (Together, Fireworks etc) don't build their own models.
Though Fireworks is one of the few providers that supports structured generation.