61–70 of 75 posts
PRs on this have been open for something like a year! I'm a bit sad about how quiet the maintainers have been on this.
Hoping to be more on top of community PRs and get them merged in the coming year.
What's the value-add compared to `outlines`? https://www.souzatharsis.com/tamingLLMs/notebooks/structured...
With the newer research - outlines/xgrammar coming out, I hope to be able to update the sampling to support more formats, increase accuracy, and improve performance.
Very annoying marketing and pretending to be anything other than just wrapper around llama.cpp.
Can you ollama haters stop with this bullshit? Does llama.cpp do dynamic model loading and unloading? Will it fetch a model you request but isn't downloaded? Does it provide SDKs? Does it have startup services it provides? There's space for things that wrap llama.cpp and solve many of its pain points. You can find piles of reports of people struggling to build and compile llama.cpp for some reason or another who then…
But this submitted link is not even about all of that, it is about what really llama.cpp does not do - it does not write more lines of marketing material than lines of code, which is that marketing material is about, lines of code that really just wrap 10x more lines of code down the line, and all of that by not making it clear as day.
Earlier quoted context omitted.
Hey! Author of the blog post here. Yes you should be able to use any model. Your mileage may vary with the smaller models but asking them to “return x in json” tends to help with accuracy (anecdotally).
Why do smaller models fail to follow? Isn't the objective of constraints that it always provides the right output type?
Earlier quoted context omitted.
The way the LLMs work is you feed them a vector, array of numbers, that represents a sequence of tokens. You turn the crank and you get a probability distribution for the next token in the sequence. You then sample the distribution to get the next token, append it to the vector, and do it again and again. Thus the typical LLM have no memory as such, it inferes what it was thinking by looking at what it has already sa…
you have described an RNN I think, don't attention heads add something that you could compare to rough &ready understanding?
In either case you can "prime it" like it was suggested.
A regular RNN has more feedback[3], like each layer feeding back to itself, as I understand it.
Happy to be corrected though.
[1]: https://jalammar.github.io/illustrated-gpt2/#one-difference-...
[2]: https://medium.com/@ikim1994914/understanding-the-modern-llm...
[3]: https://karpathy.github.io/2015/05/21/rnn-effectiveness/
If anyone needs a more powerful constrain outputs, llama.cpp support gbnf: https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
Earlier quoted context omitted.
Can you ollama haters stop with this bullshit? Does llama.cpp do dynamic model loading and unloading? Will it fetch a model you request but isn't downloaded? Does it provide SDKs? Does it have startup services it provides? There's space for things that wrap llama.cpp and solve many of its pain points. You can find piles of reports of people struggling to build and compile llama.cpp for some reason or another who then…
Sure llama.cpp does not do all of that, except that it lets you curl model from public and free to use endpoints, it does that. But SDK? - fuck that. Load, unload and startup services - who is even need that ? All this value is so minuscule compared to core functionality provided by ggml/llamacpp. But this submitted link is not even about all of that, it is about what really llama.cpp does not do - it does not write…
Has anyone seen how these constraints affect the quality of the output out of the LLM? In some instances, I'd rather parse Markdown or plain text if it means the quality of the output is higher.
There was a paper going around claiming that structured outputs did hurt the quality of the output, but it turns out their experiment setup was laughably bad [0]. It looks like, so long as you're reasonable with the prompting, you tend to get better outputs when using structure. 0. https://blog.dottxt.co/say-what-you-mean.html
Are llama.cpp and ollama leveraging llama's intrinsic structured output capability, or is this something else bolted ex-post on the output? (And if the former, how is the capability guaranteed across other models?)
Could someone explain how this is implemented? I saw on Meta's Llama page that the model has intrinsic support for structured output. My 30k ft mental model of LLM is as a text completer, so it's not clear to me how this is accomplished. Are llama.cpp and ollama leveraging llama's intrinsic structured output capability, or is this something else bolted ex-post on the output? (And if the former, how is the capability…