Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every Way to Get Structured Output from LLMs

boundaryml.com

Post reply on HN
Print view View original

21–30 of 89 posts

Re: Every Way to Get Structured Output from LLMs

#21

Post by hellovai »

StrauXX wrote: post #11

Did I understand the documentation for many of these libraries correctly in that they reprompt until they receive valid JSON? If so I don't understand why one would do that when token masking is a deterministicly verifyable way to get structured output of any kind (as done by Guidance and LMQL for instance). This is not meant to be snarky, I really am curious. Is there an upside to reprompting - aside from easier imp…

the main one is that most people don't own the model. so if you use openai / anthropic / etc then you can't use token masking. in that case, reprompting is pretty much the only option

Re: Every Way to Get Structured Output from LLMs

#22

Post by frabcus »

StrauXX wrote: post #11

Did I understand the documentation for many of these libraries correctly in that they reprompt until they receive valid JSON? If so I don't understand why one would do that when token masking is a deterministicly verifyable way to get structured output of any kind (as done by Guidance and LMQL for instance). This is not meant to be snarky, I really am curious. Is there an upside to reprompting - aside from easier imp…

My experience with models even about a year ago is that the model has firmly decided what it thinks by the time of the last layer, so the probabilities on that layer aren't very useful.

You either get the same (in this case wrong) thing differently worded, or worse you get effectively noise if the second probability is very much lower than the largest probability.

My guess is that applies here too. Better to let all the layers rethink the tokens, than force hallucination of eg a random letter when you don't expect an angle bracket

(Edit: above is assuming using logprobs and/or logit_bias with the OpenAI API, not some other masking technique)

Re: Every Way to Get Structured Output from LLMs

#23

Post by mg »

The baml config files look a lot like code. For example in baml:

    class Resume {
        name string
        education Education[] @description("Extract in the same order listed")
        skills string[] @description("Only include programming languages")
    }
Could be expressed in Python like this:

    class Resume:
        name: str
        education: List[Education] # Extract in the same order listed
        skills: List[str] # Only include programming languages
Two benefits I see are that it would make the file leaner (because Python is nicely lean) and provide free parsing and syntax highlighting.

Is there a benefit of rolling your own DSL?

Re: Every Way to Get Structured Output from LLMs

#25

Post by alhaad »

_flux wrote: post #12
alhaad wrote: post #5

Are there fine tuned models that perform better for structured / parsable outputs?

This isn't the answer to that question, but llama.cpp has a feature to constrain output to the provided grammar, such as https://github.com/ggerganov/llama.cpp/blob/master/grammars/... Others should really implement that as well. You still need to guide the model to produce e.g. JSON to get good results, but they will 100% guaranteed be valid per the grammar.

Agreed that others should implement it as well but coercing llama to output results with matching grammar needs work.

Re: Every Way to Get Structured Output from LLMs

#26

Post by martypitt »

thih9 wrote: post #15

This is an article written by BAML that shows BAML as the best. Also, BAML seems to be a commercial product with no clear pricing. > Our paid capabilities only start if you use Boundary Studio, which focuses on Monitoring, Collecting Feedback, and Improving your AI pipelines. Contact us for pricing details at contact_boundaryml.com

> Also, BAML seems to be a commercial product with no clear pricing.

You've only presented half the story. They're also Open Source (Apache 2.0), with code on github.

As you mention, some features are gated, but they seem to have a fairly solid OSS offering.

Re: Every Way to Get Structured Output from LLMs

#27

Post by _flux »

alhaad wrote: post #25
_flux wrote: post #12

Earlier quoted context omitted.

This isn't the answer to that question, but llama.cpp has a feature to constrain output to the provided grammar, such as https://github.com/ggerganov/llama.cpp/blob/master/grammars/... Others should really implement that as well. You still need to guide the model to produce e.g. JSON to get good results, but they will 100% guaranteed be valid per the grammar.

Agreed that others should implement it as well but coercing llama to output results with matching grammar needs work.

What kind of work? I've only given it a short try before moving to Ollama that doesn't have it, but it seemed to have worked there. (With ollama I need to use a retry system.)

edit: I researched a bit and apparently it can reduce performance, plus the streaming mode fails to report incorrect grammars. Overall these don't seem like deal-breakers.

Re: Every Way to Get Structured Output from LLMs

#28

Post by hellovai »

mg wrote: post #23

The baml config files look a lot like code. For example in baml: class Resume { name string education Education[] @description("Extract in the same order listed") skills string[] @description("Only include programming languages") } Could be expressed in Python like this: class Resume: name: str education: List[Education] # Extract in the same order listed skills: List[str] # Only include programming languages Two ben…

that's a great question, there's three main benefits:

1. seeing the full prompt, even though that python code feels leaner, somehow you need to convert it to a prompt. a library will do that in some way, BAML has a VSCode playground to see the entire prompt + tokenization. If we had to do this off of python/ts, we would run into the halting problem and making the playground would be much much harder.

2. there's a lot of codegen we do for users, to make life easier, e.g. w/o BAML, to now do streaming for the resume, you would have to do something like this:

class PartialResume: name: Optional[str] education: List[PartialEducation] skills: List[str]

and then at some point you need to reparse PartialResume -> Resume, we can codegen all of that for you, and give you autocomplete, type-safety for free.

3. We added a lot of static analysis / jump to definition etc to JINJA (which we use for strings), and that is much easier to navigate than f-strings.

4. Since its code-gen we can support all languages way easier, so prompting techniques in python work the same exact way for the same code in typescript.

Re: Every Way to Get Structured Output from LLMs

#29

Post by GeneralMayhem »

mg wrote: post #23

The baml config files look a lot like code. For example in baml: class Resume { name string education Education[] @description("Extract in the same order listed") skills string[] @description("Only include programming languages") } Could be expressed in Python like this: class Resume: name: str education: List[Education] # Extract in the same order listed skills: List[str] # Only include programming languages Two ben…

Two big advantages.

First, if you want a declarative config with limited, domain-specific options, rolling your own DSL instead of using something as complex as Python is much, much easier to implement. You're not actually going to be running the code either way, at least not in the normal way, and the Python syntax tree is pretty complicated.

Second, having code that looks like Python can lead your users to believe that it is in, in fact, Python. When you're doing things like using your DSL as configuration that happens at setup time, but then actually "running" the resulting config later on, that can lead to people getting themselves into trouble - for instance, they might try to use `time.now()` and end up embedding the time of the config parser as a constant in their workflow definition.

If you want to use Python as your language, you probably want to define your "DSL" as a Python library, so that you can use a normal interpreter to work with it. Maybe you have library functions that return config objects, and a user's "configuration" is an arbitrary Python file with a standard function name as an entry point. But then when you want to introspect over types, you probably need to start playing games with decorators, which is tricky again, and you have to be very careful to have that evaluation step return meaningful errors.

Starlark (https://github.com/bazelbuild/starlark) is an example of using Python-ish as a "configuration" language. That took an absolutely massive amount of engineering to get to be well-defined, and was only arguably worth it because they wanted a language that's a loop construct away from being Turing-complete. If they had wanted a basic declarative relationship language, they probably would have used textprotos or GCL.

Re: Every Way to Get Structured Output from LLMs

#30

Post by HeatrayEnjoyer »

frabcus wrote: post #22
StrauXX wrote: post #11

Did I understand the documentation for many of these libraries correctly in that they reprompt until they receive valid JSON? If so I don't understand why one would do that when token masking is a deterministicly verifyable way to get structured output of any kind (as done by Guidance and LMQL for instance). This is not meant to be snarky, I really am curious. Is there an upside to reprompting - aside from easier imp…

My experience with models even about a year ago is that the model has firmly decided what it thinks by the time of the last layer, so the probabilities on that layer aren't very useful. You either get the same (in this case wrong) thing differently worded, or worse you get effectively noise if the second probability is very much lower than the largest probability. My guess is that applies here too. Better to let all…

Why not apply it at an earlier layer then?
Post reply on HN