BAML
LLM Structured Outputs Handbook
11–20 of 67 posts
Re: LLM Structured Outputs Handbook
#12Are there output formats that are more reliable (better adherence to the schema, easier to get parse-able output) or cheaper (fewer tokens) than JSON? YAML has its own problems and TOML isn't widely adopted, but they both seem like they would be easier to generate. What have folks tried?
Yes, that's the purpose of TOON. https://github.com/toon-format/toon
Is there evidence that LLMs adhere to this format better than to JSON? I doubt that.
Re: LLM Structured Outputs Handbook
#13This is a seriously beautiful guide. I really appreciate you putting this together! I especially love the tab-through animations on the various pages, and this is one of the best explanations that I've seen. I generally feel I understand grammar-constrained generation pretty well (I've merged a handful of contributions to the llama.cpp grammar implementation), and yet I still learned some insights from your illustrat…
What does it do when the model wants to return something else, and what's better/worse about doing it in llamafile vs whatever wrapper that's calling it? How do I set retries? What if I want JSON and a range instead?
Re: LLM Structured Outputs Handbook
#14Very nicely written guide!
If the authors or readers are interested in some of the more technical details of how we optimized guidance & llguidance, we wrote up a little paper about it here: https://guidance-ai.github.io/llguidance/llg-go-brrr
Re: LLM Structured Outputs Handbook
#15I agree that building agents is basically impossible if you cannot trust the model to output valid json every time. This seems like a decent collection of the current techniques we have to force deterministic structure for production systems.
[dead]
Re: LLM Structured Outputs Handbook
#16Are there output formats that are more reliable (better adherence to the schema, easier to get parse-able output) or cheaper (fewer tokens) than JSON? YAML has its own problems and TOML isn't widely adopted, but they both seem like they would be easier to generate. What have folks tried?
I use regex to force an XML schema and then use a normal XML parser to decode.
XML is better for code, and for code parts in particular I enforce a cdata[[ part so there LLM is pretty free to do anything without escaping.
OpenAI API lets you do regex structured output and it's much better than JSON for code.
Re: LLM Structured Outputs Handbook
#17What would be the point of outputting unconstrained json if the output is consumed by a human?
Re: LLM Structured Outputs Handbook
#18This is a fantastic guide! I did a lot of work on structured generation for my PhD. Here are a few other pointers for people who might be interested:
Some libraries:
- Outlines, a nice library for structured generation
- https://github.com/dottxt-ai/outlines
- https://github.com/guidance-ai/guidance
- https://github.com/mlc-ai/xgrammar
- This one has a lot of cool technical aspects that make it an interesting project
- Efficient Guided Generation for Large Language Models
- By the outlines authors, probably the first real LLM constrained generation paper
- https://arxiv.org/abs/2307.09702
- A much more technical paper about constrained generation and implementation
- https://arxiv.org/abs/2407.08103
- A bit of self-promotion. We show where constrained generation can go wrong and discuss some techniques for the practitioner
- https://openreview.net/pdf?id=DFybOGeGDS
- Fast, High-Fidelity LLM Decoding with Regex Constraints
- Discusses adhering to the canonical tokenization (i.e., not just the constraint, but also what would be produced by the tokenizer)
- https://vivien000.github.io/blog/journal/llm-decoding-with-regex-constraints.html
- Also from the outlines team
- This is about skipping inference during constrained generation if you know there is only one valid token (common in the canonical tokenization setting)
- https://blog.dottxt.ai/coalescence.htmlRe: LLM Structured Outputs Handbook
#19I like structured outputs as much as the next guy but be careful not to try to structure natural language.
Re: LLM Structured Outputs Handbook
#20This is a fantastic guide! I did a lot of work on structured generation for my PhD. Here are a few other pointers for people who might be interested: Some libraries: - Outlines, a nice library for structured generation - https://github.com/dottxt-ai/outlines - Guidance (already covered by FlyingLawnmower in this thread), another nice library - https://github.com/guidance-ai/guidance - XGrammar, a less-featureful but…
What a gold mine!
Automata-based constraints is fun.