41–50 of 67 posts
Sure the guide presents some alternatives but they're incomparably useless VS real enforced structured output.
I get that some people will run their own models or whatever and will be able to use some of the other techniques, but that's the remaining 1%.
Stupid question but isn't this useless for 99% of users? By that I mean that either your API provider supports Structured Outputs (OpenAI and Google) or it doesn't and you're SOL. Sure the guide presents some alternatives but they're incomparably useless VS real enforced structured output. I get that some people will run their own models or whatever and will be able to use some of the other techniques, but that's the…
As they point out - this might impact results where deep reasoning is required.
So you might be better off taking the unconstrained approach with feedback.
Stupid question but isn't this useless for 99% of users? By that I mean that either your API provider supports Structured Outputs (OpenAI and Google) or it doesn't and you're SOL. Sure the guide presents some alternatives but they're incomparably useless VS real enforced structured output. I get that some people will run their own models or whatever and will be able to use some of the other techniques, but that's the…
Well, you still need to decide if structured output is the right choice. As they point out - this might impact results where deep reasoning is required. So you might be better off taking the unconstrained approach with feedback.
ESPECIALLY with situations where deep reasoning is required, since those are likely to correlate with longer JSON outputs and therefore more failure points.
This is good. It covers the two easiest dominant methods people use. It even touches on my main complaint for the one they seem to recommend. That said: - Constrained generation yields a different distribution from what a raw LLM would provide. This can be pathologically bad. My go-to example is LLMs having a preference for including ellipses in long, structured objects. Constrained generation forces closing quotes o…
Another way to do this is to use a hybrid approach. You perform unconstrained generation first, and then constrained generation on the failures.
Very nicely written guide! If the authors or readers are interested in some of the more technical details of how we optimized guidance & llguidance, we wrote up a little paper about it here: https://guidance-ai.github.io/llguidance/llg-go-brrr
This is a fantastic guide! I did a lot of work on structured generation for my PhD. Here are a few other pointers for people who might be interested: Some libraries: - Outlines, a nice library for structured generation - https://github.com/dottxt-ai/outlines - Guidance (already covered by FlyingLawnmower in this thread), another nice library - https://github.com/guidance-ai/guidance - XGrammar, a less-featureful but…
Proceeds to list all the libraries already listed in the guide.
These are cool tricks but this seems like an impedence mismatch: why would you use an LLM (a probabilistic source of plausible text) in a situation where you want a deterministic source of text where plausibility is not enough?
That might not matter to you, but it can be 2-3x slower sometimes.
This is a fantastic guide! I did a lot of work on structured generation for my PhD. Here are a few other pointers for people who might be interested: Some libraries: - Outlines, a nice library for structured generation - https://github.com/dottxt-ai/outlines - Guidance (already covered by FlyingLawnmower in this thread), another nice library - https://github.com/guidance-ai/guidance - XGrammar, a less-featureful but…