Every Way to Get Structured Output from LLMs

Structured output should not be assumed is limited to JSON. Claude performs very well with XML, as it has been trained with it, so there's no real need to put in extra work. Not XML as in conformant, schema-compliant XML, just XML as delimiters.

https://docs.anthropic.com/en/docs/build-with-claude/prompt-...

Give it examples and instructions in tags, ask it to output in tags, and force it to return early by completing for it. (Assistant:).

When you think about it, it makes a lot of sense. Even if the output is chatty, parsing it is easy because you're not looking for } which may or may not match an opening {, instead you're looking for which is much easier to parse for.

XML is also a great option, but there are a few trade offs:

> XML is a many more tokens (much slower + $$$ for complex schemas)

> regardless of if you're looking for } or its really a matter of "does your parser work". when you have three tokens that need to be correct "", the odds of a mistake are higher, instead of when you just need "}".

That said, the parser is much easier to write, we're actually considering supporting XML in BAML. have you found any reductions of accuracy?

Also, not sure if you saw this, but apparently Claude doesn't actually prefer XML, it just happens to work well with it. Was recently new info for myself as well. https://x.com/alexalbert__/status/1778550859598807178 (devrel @ Anthropic)

[Other BAML creator here!] one time we told a customer to do this to fix small json mistakes but turns out their customers don't tolerate a +20-30s increase in latency for regenerating a long json structure.

We instead had to write a parser to catch small mistakes like missing commas, quotes etc, and parse content even if there's things like reasoning in the response, like here: https://www.promptfiddle.com/Chain-of-Thought-KcSBh

Really interesting library! In the docs, could you describe in a bit more detail which kind of JSON errors it tolerates? And which models currently work best with your parsing approach?