51–60 of 75 posts
If anyone needs a more powerful constrain outputs, llama.cpp support gbnf: https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
Very annoying marketing and pretending to be anything other than just wrapper around llama.cpp.
Does llama.cpp do dynamic model loading and unloading? Will it fetch a model you request but isn't downloaded? Does it provide SDKs? Does it have startup services it provides? There's space for things that wrap llama.cpp and solve many of its pain points. You can find piles of reports of people struggling to build and compile llama.cpp for some reason or another who then clicked an Ollama installer and it worked right away.
It's also a free OSS project giving all this away, why are you being an ass and discouraging them?
If anyone needs a more powerful constrain outputs, llama.cpp support gbnf: https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
Have you found the output for arbitrary grammars to be satisfactory? My naive assumption has been that these models will produce better JSON than other formats simply by virtue of having seen so much of it.
If anyone needs a more powerful constrain outputs, llama.cpp support gbnf: https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
So I can use this with any supported models? The reason I'm asking is because I can only run 1b-3b models reliably on my hardware.
Hey! Author of the blog post here. Yes you should be able to use any model. Your mileage may vary with the smaller models but asking them to “return x in json” tends to help with accuracy (anecdotally).
PRs on this have been open for something like a year! I'm a bit sad about how quiet the maintainers have been on this.
Very annoying marketing and pretending to be anything other than just wrapper around llama.cpp.
Can you ollama haters stop with this bullshit? Does llama.cpp do dynamic model loading and unloading? Will it fetch a model you request but isn't downloaded? Does it provide SDKs? Does it have startup services it provides? There's space for things that wrap llama.cpp and solve many of its pain points. You can find piles of reports of people struggling to build and compile llama.cpp for some reason or another who then…
If anyone needs a more powerful constrain outputs, llama.cpp support gbnf: https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
How is it more powerful?
If anyone needs a more powerful constrain outputs, llama.cpp support gbnf: https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
Have you found the output for arbitrary grammars to be satisfactory? My naive assumption has been that these models will produce better JSON than other formats simply by virtue of having seen so much of it.
edit: Nm
Earlier quoted context omitted.
care to explain further? I am not sure I understand you fully
The way the LLMs work is you feed them a vector, array of numbers, that represents a sequence of tokens. You turn the crank and you get a probability distribution for the next token in the sequence. You then sample the distribution to get the next token, append it to the vector, and do it again and again. Thus the typical LLM have no memory as such, it inferes what it was thinking by looking at what it has already sa…