BAML (https://boundaryml.com/) is really great for handling structured outputs. It's what we use at my company.
LLM Structured Outputs Handbook
61–67 of 67 posts
Re: LLM Structured Outputs Handbook
#62For my use case (https://www.grokvocab.com/), I get proper JSON output without much effort. I am using Langchain4J (https://github.com/langchain4j/langchain4j) which automatically maps the output JSON to my POJO. I just prompt the model to return the output as JSON.
Re: LLM Structured Outputs Handbook
#63Earlier quoted context omitted.
> Increasing context length by complaining about schema errors is almost always worse from an end quality perspective than just retrying till the schema passes. Another way to do this is to use a hybrid approach. You perform unconstrained generation first, and then constrained generation on the failures.
There's no difference in the output distribution between always doing constrained generation and only doing it on the failures though. What's the advantage?
There's no advantage wrt output quality, but it can be more economical in some high-error regimes, with less LLM calls used in resampling (max 2 for most errors).
Re: LLM Structured Outputs Handbook
#64Earlier quoted context omitted.
There's no difference in the output distribution between always doing constrained generation and only doing it on the failures though. What's the advantage?
There's no advantage wrt output quality, but it can be more economical in some high-error regimes, with less LLM calls used in resampling (max 2 for most errors).
My point is that if you're capable of doing constrained generation and want to try once and the constrain on failure, since that has the same output distribution as doing constrained generation in the first place, you'd be better off just doing constrained generation always (max of 1 LLM call for the class of errors fixed by this).
There's only a different distribution with 2+ initial attempts before falling back to constrained, at least if I haven't screwed up any math.
Re: LLM Structured Outputs Handbook
#65To use constrained decoding like this, you really need to use an open weight model or can this be done using OpenAI, Claude, Gemini...etc?
Re: LLM Structured Outputs Handbook
#66This is good. It covers the two easiest dominant methods people use. It even touches on my main complaint for the one they seem to recommend. That said: - Constrained generation yields a different distribution from what a raw LLM would provide. This can be pathologically bad. My go-to example is LLMs having a preference for including ellipses in long, structured objects. Constrained generation forces closing quotes o…
Regarding your first point, this makes me wonder if diffusion models will be the future of constrained decoding.
Re: LLM Structured Outputs Handbook
#67This is good. It covers the two easiest dominant methods people use. It even touches on my main complaint for the one they seem to recommend. That said: - Constrained generation yields a different distribution from what a raw LLM would provide. This can be pathologically bad. My go-to example is LLMs having a preference for including ellipses in long, structured objects. Constrained generation forces closing quotes o…
Regarding your first point, this makes me wonder if diffusion models will be the future of constrained decoding.
Perhaps. Would you mind elaborating on what you're envisioning?
In both cases (auto-regressive vs diffusive), you still have some process that's being followed, and the exact steps in the process are important to the result. If you constraint at each step then you get the equivalent of something like projected gradient descent (as an analogy) and aren't guaranteed the same solution. If you constrain as a post-processing phase then (a) diffusion wasn't required for the initial generation, and (b) that's still unlikely to converge to the same distribution (for similar reasons -- using my example of ellipsis errors, if you corrected that particular mistake in post then the closest valid messages to the initial generation are likely too short and thus still incorrect).