Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLM Structured Outputs Handbook

nanonets.com

Post reply on HN
Print view View original

51–60 of 67 posts

Re: LLM Structured Outputs Handbook

#51

Post by tlarkworthy »

psadri wrote: post #37
tlarkworthy wrote: post #16

Earlier quoted context omitted.

I use regex to force an XML schema and then use a normal XML parser to decode. XML is better for code, and for code parts in particular I enforce a cdata[[ part so there LLM is pretty free to do anything without escaping. OpenAI API lets you do regex structured output and it's much better than JSON for code.

Could you share some samples / pointers on how you do this?

Yeah, this upsert_cell tool does it

https://observablehq.com/@tomlarkworthy/forking-agent#upsert...

format: { type: "grammar", syntax: "regex", definition: cellsRegex },

Where cellRegex is

cellsRegex = { const CELL_OPEN = String.raw`\s`;

  const INPUTS_BLOCK = String.raw`.*\s*`;

  const CODE_BLOCK = String.raw`\s*\s*`;

  const CELL_CLOSE = String.raw``;

  return "^(" + CELL_OPEN + INPUTS_BLOCK + CODE_BLOCK + CELL_CLOSE + ")*$";
}

And the extraction logic is here https://observablehq.com/@tomlarkworthy/robocoop-2#process

function process(content) { const doc = domParser.parseFromString( "" + content + "", "text/xml" ); const cells = [...doc.querySelectorAll("cell")]; return cells.map((cell) => { const inputsContent = cell.querySelector("inputs")?.textContent || ""; return { inputs: inputsContent.length > 0 ? inputsContent.split(",").map((s) => s.trim()) : [], code: (cell.querySelector("code")?.textContent || "").trim() }; }); }

BTW that agent is under development and not actually that good at programming. Its parent https://observablehq.com/@tomlarkworthy/robocoop-2 is actually very good at notebook programming

Re: LLM Structured Outputs Handbook

#52

Post by iamflimflam1 »

iLoveOncall wrote: post #43
iamflimflam1 wrote: post #42

Earlier quoted context omitted.

Well, you still need to decide if structured output is the right choice. As they point out - this might impact results where deep reasoning is required. So you might be better off taking the unconstrained approach with feedback.

The only "solution" with the unconstrained approach is to ask the LLM to regenerate the JSON. This is definitely more expensive than whatever downside from requesting structured outputs from the API. ESPECIALLY with situations where deep reasoning is required, since those are likely to correlate with longer JSON outputs and therefore more failure points.

Definitely. A lot of what is missing in many discussions is the absolutely essential need to have evals.

The only way to “know” what is the best (or better) approach is to have a significant number of test cases that you can measure performance against.

At the moment, for a lot of people, state of the art is “let’s try a different prompt and see if the answer on my one example is better”

Re: LLM Structured Outputs Handbook

#55

Post by max2 »

shmolyneaux wrote: post #5

Are there output formats that are more reliable (better adherence to the schema, easier to get parse-able output) or cheaper (fewer tokens) than JSON? YAML has its own problems and TOML isn't widely adopted, but they both seem like they would be easier to generate. What have folks tried?

Generating code that when ran generates JSON works well if you design builder functions thoughtfully. Takes fewer tokens too.

Re: LLM Structured Outputs Handbook

#56

Post by TomatoCo »

fragmede wrote: post #13
HanClinto wrote: post #4

This is a seriously beautiful guide. I really appreciate you putting this together! I especially love the tab-through animations on the various pages, and this is one of the best explanations that I've seen. I generally feel I understand grammar-constrained generation pretty well (I've merged a handful of contributions to the llama.cpp grammar implementation), and yet I still learned some insights from your illustrat…

What does it do when the model wants to return something else, and what's better/worse about doing it in llamafile vs whatever wrapper that's calling it? How do I set retries? What if I want JSON and a range instead?

You can't do it as part of whatever's calling it because this changes the sampler. The grammar constraints what tokens the sampler is allowed to consider, only passing tokens that are valid by the grammar.

Re: LLM Structured Outputs Handbook

#57

Post by ineedasername »

libraryofbabel wrote: post #30

Question for the well-informed people reading this thread: do SoTA models like Opus, Gemini and friends actually need output schema enforcement still, or has all the the RLVR training they do on generating code and json etc. made schema errors vanishingly unlikely? Because as a user of those models, they almost never make syntax mistakes in generating json and code; perhaps they still do output schema enforcement for…

This is going to be task-dependent, as well as limited by your (the implementer's) ability and comfort with structuring the task in solid multi-shot prompts that cover a large distribution of expected inputs, which will also help increase the ability for the model to successfully handle less common or edge case inputs-- the ones the would most typically require human-level reasoning. It can be useful to supplement this with a "tool" use for RAG lookup against a more extensive store of examples, or any time the full reference material isn't practical to dump into context. This requires thoughtful chunking.

It also requires testing. Don't think of it as a magic machine that should be able to do anything, think of it like a new employee smart enough and with enough background knowledge to do the task, if given proper job documentation. Test whether few-shot or many shot prompting works better: there's growing information about use cases where one or the other confers an advantage but so much of this is task dependent.

Consider your tolerance for errors and plan some escalation method: Hallucinations occur in part because models "have to" give an answer. Make sure that any critical cases where an error would be problematic have some way for the model to bail out with "i don't know" for human review. The first layer of escalation doesn't even have to be a human, it could be a separate model, eg Opus instead of Sonnet, or the same model but with a different setup prompt explicitly designed for handling certain cases without cluttering up context of the first one. Splitting things in this way, if there's a logical break point, is also a great way to save on token cost: If you can send only 10k of tokens in a system prompt instead of 50k and just choose which of 5 10k prompts to use for different cases then you save 80% of upstream token $$.

Consider running the model deterministic: 0 temp, same seed. It makes any errors you encounter easier to trace and debug.

Something to consider with respect to cost though: Many tasks that a SoTA can do with very little or no scaffolding can be done with these cheaper models and may not take much more scaffolding. If a SoTA giving reliable responses with zero shot prompting there's a decent chance you can save a ton of money with a flash model if you provide it one or few shot prompts. Open weight models even more so.

My anecdotal experience is that open models like Google's gemma and OpenAI's gpt-oss have behaviors more similar to their paid counterparts than other open models, making them reasonable candidates to try if you're getting good results from the paid models but they're perhaps overkill for the task.

Re: LLM Structured Outputs Handbook

#58

Post by robot-wrangler »

HanClinto wrote: post #4

This is a seriously beautiful guide. I really appreciate you putting this together! I especially love the tab-through animations on the various pages, and this is one of the best explanations that I've seen. I generally feel I understand grammar-constrained generation pretty well (I've merged a handful of contributions to the llama.cpp grammar implementation), and yet I still learned some insights from your illustrat…

> strongly believe that structured outputs are one of the most underrated features in LLM engines

Structured output is really the whole foundation of lots of our hopes and dreams. The JSONSchemaBench paper is fairly preoccupied with performance, but where it talks about quality/compliance, the "LM only" scores in the tables are pretty bad. This post highlights the on-going difficulty and confusion around doing a simple, necessary, and very routine task well.

Massaging small inputs into structured formats isn't really the point. It's about all the nontrivial cases central to MCP, tool-use, local or custom APIs. My favorite example of this is every tool-use tutorial that's pretending that "ping" accepts 2 arguments, but, it's actually more like 20 arguments with subtle gotchas. Do the tool-use demos that correctly work with 2 arguments actually work with 20? How many more retries might that take, and what does this change about the hardware and models we need for "basic" stuff?

If you had a JSON schema correctly and completely describing legal input for say ffmpeg, then the size and complexity of it would be approaching that of kubernetes schemas (where JSONBench compliance is only at .56). Can you maybe yolo generate a correct ffmpeg command without consulting any schema with SOTA models? Of course!, but that works well because ffmpeg is a well-documented tool with decades of examples floating around in the wild. What's the arg-count and type-complexity for that one important function/class in your in-house code base? For a less well-known use case or tool, if you want hallucination free and correct output, then you need structured output that works, because the alternative is rolling your own model trained on your stuff.

Re: LLM Structured Outputs Handbook

#59

Post by xboxnolifes »

bandrami wrote: post #25

These are cool tricks but this seems like an impedence mismatch: why would you use an LLM (a probabilistic source of plausible text) in a situation where you want a deterministic source of text where plausibility is not enough?

Because of their ability to handle unstructured input well.

Re: LLM Structured Outputs Handbook

#60

Post by hansvm »

vitaelabitur wrote: post #45
hansvm wrote: post #29

This is good. It covers the two easiest dominant methods people use. It even touches on my main complaint for the one they seem to recommend. That said: - Constrained generation yields a different distribution from what a raw LLM would provide. This can be pathologically bad. My go-to example is LLMs having a preference for including ellipses in long, structured objects. Constrained generation forces closing quotes o…

> Increasing context length by complaining about schema errors is almost always worse from an end quality perspective than just retrying till the schema passes. Another way to do this is to use a hybrid approach. You perform unconstrained generation first, and then constrained generation on the failures.

There's no difference in the output distribution between always doing constrained generation and only doing it on the failures though. What's the advantage?
Post reply on HN