Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every Way to Get Structured Output from LLMs

boundaryml.com

Post reply on HN
Print view View original

81–89 of 89 posts

Re: Every Way to Get Structured Output from LLMs

#81

Post by resiros »

Jayakumark wrote: post #74
resiros wrote: post #72

I expected to read about the methods used by the libraries to get the structured output and not a comparison of the language compatibility for each. Fortunately the same author have a blog post ( https://www.boundaryml.com/blog/type-definition-prompting-ba... ) explaining how their approach works and how it compares to instructor ( https://github.com/jxnl/instructor ). Basically these libraries provide two things: 1.…

Will you be able to share an example code or gist ?

If you look into the instructor code(https://github.com/jxnl/instructor/blob/06a49e7824729b8df1f7...). Here is the core code snippet they use:

            message = dedent(
                f"""
                As a genius expert, your task is to understand the content and provide
                the parsed objects in json that match the following json_schema:\n

                {json.dumps(response_model.model_json_schema(), indent=2)}

                Make sure to return an instance of the JSON, not the schema itself
                """
            )
Then depending on the mode, either they add another message `Return the correct JSON response within a ```json codeblock. not the JSON_SCHEMA` or they set the response format to json.

Re: Every Way to Get Structured Output from LLMs

#82

Post by aaronvg »

b2v wrote: post #60
hellovai wrote: post #57

Earlier quoted context omitted.

nothing specific, but you can try our prompt / datamodel out on https://www.promptfiddle.com or if you're open to share your prompt / data model with, I can send over my best guess of a good prompt! We've found these models works even with over 50+ fields / nested and whatnot decently well!

I might share it with you later on your discord server. > I can send over my best guess of a good prompt! Now if you could automate the above process by "fitting" a first draft prompt to a wanted schema, ie where your library makes a few adjustments if some assertions do not pass by have having a chat of its own with the LLM, that would be super useful! Heck i might just implement it myself.

[Another BAML creator here]. I agree this is an interesting direction! We have a "chat" feature on our roadmap to do this right in the VSCode playground, where an AI agent will have context on your prompt, schema, (and baml test results etc) and help you iterate on the prompt automatically. We've done this before and have been surprised by how good the LLM feedback can be.

We just need a bit better testing flow within BAML since we do not support adding assertions just yet.

Re: Every Way to Get Structured Output from LLMs

#85

Post by StrauXX »

hellovai wrote: post #21
StrauXX wrote: post #11

Did I understand the documentation for many of these libraries correctly in that they reprompt until they receive valid JSON? If so I don't understand why one would do that when token masking is a deterministicly verifyable way to get structured output of any kind (as done by Guidance and LMQL for instance). This is not meant to be snarky, I really am curious. Is there an upside to reprompting - aside from easier imp…

the main one is that most people don't own the model. so if you use openai / anthropic / etc then you can't use token masking. in that case, reprompting is pretty much the only option

It does. With OpenAI at least you definetly can use token masking. There are some limitations but even those are circumventable. I have used token masking on the OpenAI API with LMQL without any issues.

Re: Every Way to Get Structured Output from LLMs

#86

Post by joatmon-snoo »

gavindean90 wrote: post #78
hellovai wrote: post #6

Earlier quoted context omitted.

XML is also a great option, but there are a few trade offs: > XML is a many more tokens (much slower + $$$ for complex schemas) > regardless of if you're looking for } or its really a matter of "does your parser work". when you have three tokens that need to be correct " ", the odds of a mistake are higher, instead of when you just need "}". That said, the parser is much easier to write, we're actually considering su…

I think once you have The ” ensures that our “output” string is ended. In JSON all of that semantic meaning get put into the one token }.

Hmm, that's an interesting way of thinking about it. The way I see it, I trust XML less, because the sparser representation gives it more room to make a mistake: if you think of every token as an opportunity to be correct or wrong, the higher token count needed to represent content in XML gives the model a higher chance to get the output wrong (kinda like the birthday paradox).

(Plus, more output tokens is more expensive!)

e.g.

using the cl_100k tokenizer (what GPT4 uses), this JSON is 60 tokens:

    {
      "method": "GET",
      "endpoint": "/api/model/details",
      "headers": {
        "Authorization": "Bearer YOUR_ACCESS_TOKEN",
        "Content-Type": "application/json"
      },
      "queryParams": {
        "model_id": "12345"
      }
    }
whereas this XML is 76 tokens:

    
    GET
    /api/model/details
    
        Bearer YOUR_ACCESS_TOKEN
        application/json
    
    
        12345
You can check out the tokenization here by toggling "show tokens": https://www.promptfiddle.com/json-vs-xml-token-count-BtXe3

Re: Every Way to Get Structured Output from LLMs

#87

Post by tarasglek »

joatmon-snoo wrote: post #86
gavindean90 wrote: post #78

Earlier quoted context omitted.

I think once you have The ” ensures that our “output” string is ended. In JSON all of that semantic meaning get put into the one token }.

Hmm, that's an interesting way of thinking about it. The way I see it, I trust XML less, because the sparser representation gives it more room to make a mistake: if you think of every token as an opportunity to be correct or wrong, the higher token count needed to represent content in XML gives the model a higher chance to get the output wrong (kinda like the birthday paradox). (Plus, more output tokens is more expen…

you will love yaml since its a similar improvement in token use over json

Re: Every Way to Get Structured Output from LLMs

#88

Post by vladsanchez »

resiros wrote: post #80
vladsanchez wrote: post #77

Earlier quoted context omitted.

https://agenta.ai/

Thanks for sharing the link, but no agenta is not a library that can help with getting structured outputs from LLMs (at least not in the way discussed in the parent comment). It's a prompt management, evaluation and observability platform for LLM apps.

Just stumbled with https://controlflow.ai/ today. Perhaps it serves to structure outputs as "agentic" workflows in pursuit of LLM autonomy.

Let us know your opinion.

Post reply on HN