Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Structured Outputs with Ollama

ollama.com

Post reply on HN
Print view View original

11–20 of 75 posts

Re: Structured Outputs with Ollama

#11

Post by parthsareen »

hackernewds wrote: post #6
parthsareen wrote: post #5

Earlier quoted context omitted.

We’ve been keeping a close eye on this as well as research is coming out. We’re looking into improving sampling as a whole on both speed and accuracy. Hopefully with those changes we might also enable general structure generation not only limited to JSON.

Who is "we"?

I authored the blog with some other contributors and worked on the feature (PR: https://github.com/ollama/ollama/pull/7900).

The current implementation uses llama.cpp GBNF grammars. The more recent research (Outlines, XGrammar) points to potentially speeding up the sampling process through FSTs and GPU parallelism.

Re: Structured Outputs with Ollama

#12

Post by thot_experiment »

bluechair wrote: post #2

Has anyone seen how these constraints affect the quality of the output out of the LLM? In some instances, I'd rather parse Markdown or plain text if it means the quality of the output is higher.

YMMV, it's a negative effect in terms of "reasoning" but the delta isn't super significant in most cases. It really depends on the LLM and whether your prompt is likely to generate a JSON response to begin with, the more you have to coerce the LLM the less likely it is to generate sane input. With smaller models you more quickly end up at the edge of space where the LLM has meaningful predictive power and so the outputs start getting closer to random noise.

FWIW measured by me using a vibes based method, nothing rigorous just a lot of hours spent on various LLM projects. I have not used these particular tools yet but ollama was previously able to guarantee json output through what I assume is similar techniques and my partner and I worked previously on a jsonformer-like thing for oobabooga, another LLM runtime tool.

Re: Structured Outputs with Ollama

#13

Post by lolinder »

bluechair wrote: post #2

Has anyone seen how these constraints affect the quality of the output out of the LLM? In some instances, I'd rather parse Markdown or plain text if it means the quality of the output is higher.

Working with OpenAI's models I've found a very good strategy is to have two passes if you can afford the extra tokens: one pass uses a heavy model and natural language with markdown sections discussing the reasoning and providing a final natural language answer (ideally labeled clearly with a markdown header). The second pass can use a cheaper and faster model to put the answer into a structured output format for consumption by the non-LLM parts of the pipeline.

You basically use JSON schema mode to draw a clean boundary around the wishy-washy language bits, using the LLM as a preprocessor to capture its own output in a useful format.

Re: Structured Outputs with Ollama

#14

Post by chirau »

This is wonderful news.

I was actually scratching my head on how to structure a regular prompt to produce csv data without extra nonsense like "Here is your data" and "Please note blah blah" at the beginning and end, so this is much welcome as I can define exactly what I want returned then just push structured output to csv.

Re: Structured Outputs with Ollama

#15

Post by xnx »

Is there a best approach for providing structured input to LLMs? Example: feed in 100 sentences and get each one classified in different ways. It's easy to get structured data out, but my approach of prefixing line numbers seems clumsy.

Re: Structured Outputs with Ollama

#17

Post by firejake308 »

chirau wrote: post #14

This is wonderful news. I was actually scratching my head on how to structure a regular prompt to produce csv data without extra nonsense like "Here is your data" and "Please note blah blah" at the beginning and end, so this is much welcome as I can define exactly what I want returned then just push structured output to csv.

Remember that you still need to include an instruction to produce a CSV to get the prompt into the right context to generate a CSV that makes sense. Otherwise, you may get output that is technically in the CSV format but doesn't make any sense because the model was actually trying to write a paragraph response and the token sampler just selected really low-probability tokens that the model didn't really want to say.

Re: Structured Outputs with Ollama

#18

Post by netghost »

parthsareen wrote: post #11
hackernewds wrote: post #6

Earlier quoted context omitted.

Who is "we"?

I authored the blog with some other contributors and worked on the feature (PR: https://github.com/ollama/ollama/pull/7900 ). The current implementation uses llama.cpp GBNF grammars. The more recent research (Outlines, XGrammar) points to potentially speeding up the sampling process through FSTs and GPU parallelism.

Thank you for the details!

Re: Structured Outputs with Ollama

#20

Post by thot_experiment »

lxe wrote: post #10

I'm still running oobabooga because of its exlv2 support which does much more efficient inference on dual 3090s

I haven't touched ooba in a while, what's the situation like with exl2 vs the non-homogeneous quantization methods people are using like q3k_s or whatever. IIRC while exl2 is faster the gptq quants were outperforming it in terms of accuracy esp at lower bit depths.
Post reply on HN