PRs on this have been open for something like a year! I'm a bit sad about how quiet the maintainers have been on this.
Yeah... I was hoping to collaborate in building this but the conversation reached an abrupt end.
Structured Outputs with Ollama
71–75 of 75 posts
Re: Structured Outputs with Ollama
#72What's the value-add compared to `outlines`? https://www.souzatharsis.com/tamingLLMs/notebooks/structured...
Hey! Author of the blog here. The current implementation uses llama.cpp GBNF which has allowed for a quick implementation. The biggest value-add at this time was getting the feature out. With the newer research - outlines/xgrammar coming out, I hope to be able to update the sampling to support more formats, increase accuracy, and improve performance.
[deleted]
Re: Structured Outputs with Ollama
#73What's the value-add compared to `outlines`? https://www.souzatharsis.com/tamingLLMs/notebooks/structured...
Hey! Author of the blog here. The current implementation uses llama.cpp GBNF which has allowed for a quick implementation. The biggest value-add at this time was getting the feature out. With the newer research - outlines/xgrammar coming out, I hope to be able to update the sampling to support more formats, increase accuracy, and improve performance.
Hi, just wanted to say how much I appreciate your work.
I'm curious if you have considered implementing Microsoft's Guidance (https://github.com/guidance-ai/guidance)? Their approach offers significant speed improvements, which I understand can sometimes be shortcoming of GBNF (e.g https://github.com/ggerganov/llama.cpp/issues/4218).
Re: Structured Outputs with Ollama
#74Could someone explain how this is implemented? I saw on Meta's Llama page that the model has intrinsic support for structured output. My 30k ft mental model of LLM is as a text completer, so it's not clear to me how this is accomplished. Are llama.cpp and ollama leveraging llama's intrinsic structured output capability, or is this something else bolted ex-post on the output? (And if the former, how is the capability…
presumably at each step they mask out all tokens that would be invalid at that step according to the grammar
That makes sense. Thanks
Re: Structured Outputs with Ollama
#75Earlier quoted context omitted.
Hey! Author of the blog here. The current implementation uses llama.cpp GBNF which has allowed for a quick implementation. The biggest value-add at this time was getting the feature out. With the newer research - outlines/xgrammar coming out, I hope to be able to update the sampling to support more formats, increase accuracy, and improve performance.
Hi, just wanted to say how much I appreciate your work. I'm curious if you have considered implementing Microsoft's Guidance ( https://github.com/guidance-ai/guidance )? Their approach offers significant speed improvements, which I understand can sometimes be shortcoming of GBNF (e.g https://github.com/ggerganov/llama.cpp/issues/4218 ).
Yes! I have checked guidance out, as well as a few others. Planning to refactor sampling in the near future which would include improving using grammars for sampling as well. Thanks for sharing!