41–50 of 99 posts
Google's Gemini API is a bit odd with structured outputs. If you specify an Application/JSON response mimetype, it will reliably respond with a consistent JSON output without any prompt engineering shenanigans. For my workflows, this setting plus providing a JSON Schema in the system prompt works even with complex schema. The Gemini API has a canonical implementation of structured outputs where you can instead pass t…
Hmm, so if structured output affects the quality of the response maybe it's better to convert the output to a structured format as a post-processing step?
[1]: https://app.sqlai.ai
Google's Gemini API is a bit odd with structured outputs. If you specify an Application/JSON response mimetype, it will reliably respond with a consistent JSON output without any prompt engineering shenanigans. For my workflows, this setting plus providing a JSON Schema in the system prompt works even with complex schema. The Gemini API has a canonical implementation of structured outputs where you can instead pass t…
It's still baffling to me that the various API providers don't let us upload our custom grammars. It would enable so many use cases, like HTML generation for example, at essentially no cost on their part.
OpenAI has started to (at least for tool calls): https://platform.openai.com/docs/guides/function-calling#con...
This is a great writeup! There was a period where reliable structured output was a significant differentiator and was the 'secret sauce' behind some companies success. A NL->SQL company I am familiar with comes to mind. Nice to see this both public and supported by a growing ecosystem of libraries. One statement surprised me was that the author thinks "models over time will just be able to output JSON perfectly witho…
[dead]
We actually don’t like constrained generation as approach - among other issues it limits your ability to use reasoning - and instead the technique we’re using is algorithm-driven error-tolerant output parsing.
It's still baffling to me that the various API providers don't let us upload our custom grammars. It would enable so many use cases, like HTML generation for example, at essentially no cost on their part.
When you say custom grammar, do you mean something other than a JSON schema, because they support that?
I've a related observation. In my experience the amount of hallucinated urls with structured output (think of a field `url` or `link`) is pretty high. Especially compared to the alternative approach, where you let the llm generate text and then use a second llm to convert the text into the desired structured format.
With structured output, it's like the llm is forced to answer in a very specific way. So if there is no url for the given field, it makes up the url.
Here a related quote from the article:
> Structured outputs builds on top of sampling by constraining the model's output to a specific format.
Earlier quoted context omitted.
I'm pretty sure the grammar is generated from the Json schema, it doesn't just constrain json syntax, it constraints on the schema (including enums and such). The schema is also given to the model (at least in openai) you can put instructions in the json schema as well that will be taken into account.
Perhaps I worded that poorly. What I mean by semantic correctness is that the model could output nonsensical values for some things. Say in a game, "normal" health is ~100hp and the model creates a wizard with 50hp but then a mouse with 10000hp. So you're guaranteed to get a parsable json object (syntactically correct) but what the values are in that json is not guaranteed to make sense in the given context.
I spent a couple years building a high performance, expressive library for structured outputs in LLMs. Our library is used by OpenAI for structured outputs on the hosted API. Happy to answer questions on how this works: User friendly library that connects to lots of OSS model serving backends: https://github.com/guidance-ai/guidance/ Core Rust library written for high performance mask computation (written mostly by m…
I'm yet to see a thorough comparison of design, performance and reliability between these options (along with outlines etc)