Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every Way to Get Structured Output from LLMs

boundaryml.com

Post reply on HN
Print view View original

11–20 of 89 posts

Re: Every Way to Get Structured Output from LLMs

#11

Post by StrauXX »

Did I understand the documentation for many of these libraries correctly in that they reprompt until they receive valid JSON? If so I don't understand why one would do that when token masking is a deterministicly verifyable way to get structured output of any kind (as done by Guidance and LMQL for instance). This is not meant to be snarky, I really am curious. Is there an upside to reprompting - aside from easier implementation.

Re: Every Way to Get Structured Output from LLMs

#12

Post by _flux »

alhaad wrote: post #5

Are there fine tuned models that perform better for structured / parsable outputs?

This isn't the answer to that question, but llama.cpp has a feature to constrain output to the provided grammar, such as https://github.com/ggerganov/llama.cpp/blob/master/grammars/...

Others should really implement that as well. You still need to guide the model to produce e.g. JSON to get good results, but they will 100% guaranteed be valid per the grammar.

Re: Every Way to Get Structured Output from LLMs

#14

Post by hellovai »

JakaJancar wrote: post #9

AI noob question: Why do OpenAI/Anthropic/... not support constraining token generation? I'd imagine producing valid structured output would be at the top of their feature request lists.

not a noob question, here's how the LLM works:

```

prompt = "..."

output = []

do:

  token_probabilities = call_model(prompt)

  best_token = pick_best(token_probabilities)

  if best_token == '':

    break

  output += best_token
while true

return output

```

basically to support generation they would need to modify pick_best to support constraining. That would make it so they can't optimize the hot loop at their scales. They support super broad output constraints like JSON which apply to everyone, but that leads to other issues (things like chain-of-thought/reasoning perform way worse in structured responses).

Re: Every Way to Get Structured Output from LLMs

#15

Post by thih9 »

This is an article written by BAML that shows BAML as the best.

Also, BAML seems to be a commercial product with no clear pricing.

> Our paid capabilities only start if you use Boundary Studio, which focuses on Monitoring, Collecting Feedback, and Improving your AI pipelines. Contact us for pricing details at contact_boundaryml.com

Re: Every Way to Get Structured Output from LLMs

#16

Post by hellovai »

dsign wrote: post #10
hellovai wrote: post #3

Hey everyone! One of the creators of BAML here! Appreciate sharing this post. For anyone interested in playing around with an interactive version of BAML online, check it out here: https://www.promptfiddle.com

Really interesting library! In the docs, could you describe in a bit more detail which kind of JSON errors it tolerates? And which models currently work best with your parsing approach?

Thanks! We should add that to the docs haha. But the here's a few:

- keys without strings

- coercing singular types -> arrays when the response requires an array

- removing any prefix or suffix tags

- picking the best of many JSON candidates in a string

- unescaped newlines + quotes so "afds"asdf" converts to "afds\"asdf"

In terms of models, honestly, we tried as bad as llama2, and it seems to work in quite a few use cases

Re: Every Way to Get Structured Output from LLMs

#18

Post by dsign »

hellovai wrote: post #16
dsign wrote: post #10

Earlier quoted context omitted.

Really interesting library! In the docs, could you describe in a bit more detail which kind of JSON errors it tolerates? And which models currently work best with your parsing approach?

Thanks! We should add that to the docs haha. But the here's a few: - keys without strings - coercing singular types -> arrays when the response requires an array - removing any prefix or suffix tags - picking the best of many JSON candidates in a string - unescaped newlines + quotes so "afds"asdf" converts to "afds\"asdf" In terms of models, honestly, we tried as bad as llama2, and it seems to work in quite a few use…

Thanks! I see myself using the library soon :-)

Re: Every Way to Get Structured Output from LLMs

#19

Post by hellovai »

thih9 wrote: post #15

This is an article written by BAML that shows BAML as the best. Also, BAML seems to be a commercial product with no clear pricing. > Our paid capabilities only start if you use Boundary Studio, which focuses on Monitoring, Collecting Feedback, and Improving your AI pipelines. Contact us for pricing details at contact_boundaryml.com

our paid product is still in Beta actually as we're continuing to build it out, but BAML itself is and always will be open source (runs fully locally as well - no extra network calls).

in terms of parsing, I do think we're likely the best approach as of now. Most other libraries do reprompting or rely on constraining grammars which require owning the model. Reprompting = slow + $$, constraining grammars = require owning the model. we just tried a new approach: parse the output in a more clever way.

Post reply on HN