Structured Outputs with Ollama • Hacker News Board

Re: Structured Outputs with Ollama

#31

Post by pamelafox » Sat, Dec 07, 2024, 5:10 AM UTC

parthsareen wrote: post #4

quaintdev wrote: post #3

So I can use this with any supported models? The reason I'm asking is because I can only run 1b-3b models reliably on my hardware.

Hey! Author of the blog post here. Yes you should be able to use any model. Your mileage may vary with the smaller models but asking them to “return x in json” tends to help with accuracy (anecdotally).

Do you happen to know if got-4o would be negatively affected by the addition of “return x in json”? I’m debating whether I could use the same prompt across all models, hosted and ollama.

Re: Structured Outputs with Ollama

#32

Post by JackYoustra » Sat, Dec 07, 2024, 5:55 AM UTC

PRs on this have been open for something like a year! I'm a bit sad about how quiet the maintainers have been on this.

Re: Structured Outputs with Ollama

#33

Post by ein0p » Sat, Dec 07, 2024, 6:49 AM UTC

That's very useful. To see why, try to get an LLM _reliably_ generate JSON output without this. Sometimes it will, but sometimes it'll just YOLO and produce something you didn't ask for, that can't be parsed.

Re: Structured Outputs with Ollama

#34

Post by guerrilla » Sat, Dec 07, 2024, 7:12 AM UTC

No way. This is amazing and one of the things I actually wanted. I love ollama be because it makes using an LLM feel like using any other UNIX program. It makes LLMs feel like they belong on UNIX.

Question though. Has anyone had luck running it on AMD GPUs? I've heard it's harder but I really want to support the competition when I get cards next year.

Re: Structured Outputs with Ollama

#35

Post by bonzini » Sat, Dec 07, 2024, 7:21 AM UTC

chirau wrote: post #30

KTibow wrote: post #29

Earlier quoted context omitted.

A lot of the time you can prevent this by prefilling the output with ```\n and stopping at ```.

care to explain further? I am not sure I understand you fully

``` is markdown for pre formatted text. It puts the LLM in the mood of generating machine readable data instead of prose.

Re: Structured Outputs with Ollama

#36

Post by lormayna » Sat, Dec 07, 2024, 7:24 AM UTC

This is a fantastic news! I spent hours on fine tuning my prompt to summarise text and output in JSON and still have some issues sometimes. Is this feature available also with Go?

Re: Structured Outputs with Ollama

#37

Post by jimmySixDOF » Sat, Dec 07, 2024, 8:33 AM UTC

rdescartes wrote: post #23

If anyone needs a more powerful constrain outputs, llama.cpp support gbnf: https://github.com/ggerganov/llama.cpp/blob/master/grammars/...

Thats is exactly what they are using

Re: Structured Outputs with Ollama

#38

Post by rcarmo » Sat, Dec 07, 2024, 9:53 AM UTC

I must say it is nice to see the curl example first. As much as I like Pydantic, I still prefer to hand-code the schemas, since it makes it easier to move my prototypes to Go (or something else).

Re: Structured Outputs with Ollama

#39

Post by rcarmo » Sat, Dec 07, 2024, 9:55 AM UTC

guerrilla wrote: post #34

No way. This is amazing and one of the things I actually wanted. I love ollama be because it makes using an LLM feel like using any other UNIX program. It makes LLMs feel like they belong on UNIX. Question though. Has anyone had luck running it on AMD GPUs? I've heard it's harder but I really want to support the competition when I get cards next year.

Yes, even on iGPUs. I was running it fairly well on a mini-PC with a 780M and the BIOS a set to allocate 16GB of shared memory to it.

Re: Structured Outputs with Ollama

#40

Post by jstanley » Sat, Dec 07, 2024, 10:10 AM UTC

chirau wrote: post #30

KTibow wrote: post #29

Earlier quoted context omitted.

A lot of the time you can prevent this by prefilling the output with ```\n and stopping at ```.

care to explain further? I am not sure I understand you fully

A lot of the time you can prevent this by prefilling the output with "```\n", and stopping at "```".