Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Structured Outputs with Ollama

ollama.com

Post reply on HN
Print view View original

1–10 of 75 posts

Re: Structured Outputs with Ollama

#4

Post by parthsareen »

quaintdev wrote: post #3

So I can use this with any supported models? The reason I'm asking is because I can only run 1b-3b models reliably on my hardware.

Hey! Author of the blog post here. Yes you should be able to use any model. Your mileage may vary with the smaller models but asking them to “return x in json” tends to help with accuracy (anecdotally).

Re: Structured Outputs with Ollama

#5

Post by parthsareen »

bluechair wrote: post #2

Has anyone seen how these constraints affect the quality of the output out of the LLM? In some instances, I'd rather parse Markdown or plain text if it means the quality of the output is higher.

We’ve been keeping a close eye on this as well as research is coming out. We’re looking into improving sampling as a whole on both speed and accuracy.

Hopefully with those changes we might also enable general structure generation not only limited to JSON.

Re: Structured Outputs with Ollama

#6

Post by hackernewds »

parthsareen wrote: post #5
bluechair wrote: post #2

Has anyone seen how these constraints affect the quality of the output out of the LLM? In some instances, I'd rather parse Markdown or plain text if it means the quality of the output is higher.

We’ve been keeping a close eye on this as well as research is coming out. We’re looking into improving sampling as a whole on both speed and accuracy. Hopefully with those changes we might also enable general structure generation not only limited to JSON.

Who is "we"?

Re: Structured Outputs with Ollama

#7

Post by nikolayasdf123 »

bluechair wrote: post #2

Has anyone seen how these constraints affect the quality of the output out of the LLM? In some instances, I'd rather parse Markdown or plain text if it means the quality of the output is higher.

same here. I noticed that when you ask model to generate elaborate responses in natural text, and then come up with an answer, quality is orders of magnitude better, and something in line you would expect human-like reasoning.

asking LLM to directly generate JSON gives much worser results, similar to either random guess or intuition.

Re: Structured Outputs with Ollama

#8

Post by benreesman »

bluechair wrote: post #2

Has anyone seen how these constraints affect the quality of the output out of the LLM? In some instances, I'd rather parse Markdown or plain text if it means the quality of the output is higher.

I can say that I was categorically wrong about the utility of things like instructor.

It’s easy to burn a lot of tokens but if the thing you’re doing merits the cost? You can be a bully with it and while its never the best, 95% as good for zero effort is a tool in one’s kit.

Post reply on HN