Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every Way to Get Structured Output from LLMs

boundaryml.com

Post reply on HN
Print view View original

61–70 of 89 posts

Re: Every Way to Get Structured Output from LLMs

#61

Post by michaelt »

hellovai wrote: post #21
StrauXX wrote: post #11

Did I understand the documentation for many of these libraries correctly in that they reprompt until they receive valid JSON? If so I don't understand why one would do that when token masking is a deterministicly verifyable way to get structured output of any kind (as done by Guidance and LMQL for instance). This is not meant to be snarky, I really am curious. Is there an upside to reprompting - aside from easier imp…

the main one is that most people don't own the model. so if you use openai / anthropic / etc then you can't use token masking. in that case, reprompting is pretty much the only option

In the specific cases of openai and anthropic, both have 'tool use' interfaces which will generate valid JSON following a schema of your choice.

You're right, though, that reprompting works with pretty much everything out there, including hosted models that don't have tool use as part of their API. And its simple too, you don't even need to know what "token masking" is.

Reprompting can also apply arbitrarily criteria that are more complex than just a json schema. You ask it to choose an excerpt of a document and the string it returns isn't an excerpt? Just reprompt.

Re: Every Way to Get Structured Output from LLMs

#64

Post by jari_mustonen »

A half year ago (a long time, I know), I tried to get structured answers from GPT-4. The structure was not complex, but I needed to extract a specific answer like "Please identify and categorize the following text as A or B" or "Please grade the following text on criteria A on a scale from 1 to 10".

First, I noticed that enforcing a JSON format on output generally lowered the quality of the results. Referring to JSON seemed to primed the LLM to be more "programmatical."

Second, I noticed that forcing LLM to answer with a single word is next to impossible. It won't do it consistently, and generally, it lowers quality.

Here's what I eventually learned: Markdown is a machine-readable enough for post-processing and easy output format for LLMs. I give the structure (a list of headings) for the LLM, which conforms to them 100% of the time. I always had a section called "Final Comments" where the LLM can blather away the things that it sometimes just needs to say after giving the answer. This can be then ignored when parsing the answer.

Also, it is good to understand that LLMs do better when you allow them to "think aloud." This Markdown output is good for that.

Re: Every Way to Get Structured Output from LLMs

#65

Post by LeifCarrotson »

jari_mustonen wrote: post #64

A half year ago (a long time, I know), I tried to get structured answers from GPT-4. The structure was not complex, but I needed to extract a specific answer like "Please identify and categorize the following text as A or B" or "Please grade the following text on criteria A on a scale from 1 to 10". First, I noticed that enforcing a JSON format on output generally lowered the quality of the results. Referring to JSON…

> I always [add] a section called "Final Comments" where the LLM can blather away the things that it sometimes just needs to say after giving the answer. This can [then be] ignored when parsing the answer.

This is a great tip for gathering data from engineers too. But maybe don't say it will be ignored out loud. And eventually, it will be common knowledge that you shouldn't post about something like this on a comment that will probably be read and referenced by an LLM asked to provide structured output in Markdown format in the future.

    ...
 
    [Criteria A Score: 7]
    The writing contained...

    [Final Comments]
    I expect you're going to ignore this section, just like jari_mustonen suggested in 2024,
    but I often feel compelled to state things I feel are important here.
    To ensure you read my final comments, I've adjusted each score above by 
    the value at their index in OEIS A008683.

Re: Every Way to Get Structured Output from LLMs

#66

Post by anonymoushn »

HeatrayEnjoyer wrote: post #30
frabcus wrote: post #22

Earlier quoted context omitted.

My experience with models even about a year ago is that the model has firmly decided what it thinks by the time of the last layer, so the probabilities on that layer aren't very useful. You either get the same (in this case wrong) thing differently worded, or worse you get effectively noise if the second probability is very much lower than the largest probability. My guess is that applies here too. Better to let all…

Why not apply it at an earlier layer then?

At earlier layers the activations don't correspond as cleanly to tokens, and I expect that vendor APIs for proprietary LLMs wouldn't let you do anything like this.

Re: Every Way to Get Structured Output from LLMs

#67

Post by infecto »

jari_mustonen wrote: post #64

A half year ago (a long time, I know), I tried to get structured answers from GPT-4. The structure was not complex, but I needed to extract a specific answer like "Please identify and categorize the following text as A or B" or "Please grade the following text on criteria A on a scale from 1 to 10". First, I noticed that enforcing a JSON format on output generally lowered the quality of the results. Referring to JSON…

You are spot on and this has been with most/all LLM since the beginning.

- Asking for structured output in the same request as the main unit of work introduces the chance of lower quality output. Instead you should be doing your unit of work and then following up with a different request/model to parse it into json or your flavor of structure.

- Grading on numerical scales introduces weird bias. I never went down the route too far but I would notice certain floating point numbers would show up too often when using numerical scales. Using a word based scale works a lot better.

Re: Every Way to Get Structured Output from LLMs

#68

Post by simonw »

"Constrained streaming generation produces partial objects, but no good ways of interacting with the partial objects, since they are not yet parse-able."

I've successfully used the ijson Python streaming JSON parser for this, notes here: https://til.simonwillison.net/json/ijson-stream

Re: Every Way to Get Structured Output from LLMs

#69

Post by simonw »

jari_mustonen wrote: post #64

A half year ago (a long time, I know), I tried to get structured answers from GPT-4. The structure was not complex, but I needed to extract a specific answer like "Please identify and categorize the following text as A or B" or "Please grade the following text on criteria A on a scale from 1 to 10". First, I noticed that enforcing a JSON format on output generally lowered the quality of the results. Referring to JSON…

I've found the most effective trick for this is to use examples. With the messages array format you can even "fake" previous interactions to provide examples of what you want to happen - send in several prompt / example-response pairs and most models will get the idea pretty quickly.
Post reply on HN