Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The local LLM ecosystem doesn’t need Ollama

sleepingrobots.com

Post reply on HN
Print view View original

21–30 of 222 posts

Re: The local LLM ecosystem doesn’t need Ollama

#21

Post by utopiah »

Not sure why VLC doesn't do that.

It's a joke... but also not really? I mean VLC is "just" an interface to play videos. Videos are content files one "interact" with, mostly play/pause and few other functions like seeking. Because there are different video formats VLC relies on codecs to decode the videos, so basically delegating the "hard" part to codecs.

Now... what's the difference here? A model is a codec, the interactions are sending text/image/etc to it, output is text/image/etc out. It's not even radically bigger in size as videos can be huge, like models.

I'm confused as why this isn't a solved problem, especially (and yes I'm being a big sarcastic here, can't help myself) in a time where "AI" supposedly made all smart wise developers who rely on it 10x or even 1000x more productive.

Weird.

Re: The local LLM ecosystem doesn’t need Ollama

#22

Post by sudb »

utopiah wrote: post #21

Not sure why VLC doesn't do that. It's a joke... but also not really? I mean VLC is "just" an interface to play videos. Videos are content files one "interact" with, mostly play/pause and few other functions like seeking. Because there are different video formats VLC relies on codecs to decode the videos, so basically delegating the "hard" part to codecs. Now... what's the difference here? A model is a codec, the int…

What problem is it that you are confused isn't solved?

I think the codec analogy is neat but isn't the codec here llama.cpp, and the models are content files? Then the equivalent of VLC are things like LMStudio etc. which use llama.cpp to let you run models locally?

I'd guess one reason we haven't solved the "codec" layer is that there doesn't seem to be a standard that open model trainers have converged on yet?

Re: The local LLM ecosystem doesn’t need Ollama

#23

Post by pheggs »

speedgoose wrote: post #12

I prefer Ollama over the suggested alternatives. I will switch once we have good user experience on simple features. A new model is released on HF or the Ollama registry? One `ollama pull` and it's available. It's underwhelming? `ollama rm`.

you can pull directly from huggingface with llama.cpp, and it also has a decent web chat included

Re: The local LLM ecosystem doesn’t need Ollama

#24

Post by speedgoose »

pheggs wrote: post #23
speedgoose wrote: post #12

I prefer Ollama over the suggested alternatives. I will switch once we have good user experience on simple features. A new model is released on HF or the Ollama registry? One `ollama pull` and it's available. It's underwhelming? `ollama rm`.

you can pull directly from huggingface with llama.cpp, and it also has a decent web chat included

Does it have a model registry with an API and hot swapping or you still have to use sometime like llama swap as suggested in the article ? Or is it CLI?

Re: The local LLM ecosystem doesn’t need Ollama

#25

Post by cientifico »

For most users that wanted to run LLM locally, ollama solved the UX problem.

One command, and you are running the models even with the rocm drivers without knowing.

If llama provides such UX, they failed terrible at communicating that. Starting with the name. Llama.cpp: that's a cpp library! Ollama is the wrapper. That's the mental model. I don't want to build my own program! I just want to have fun :-P

Re: The local LLM ecosystem doesn’t need Ollama

#26

Post by speedgoose »

kennywinker wrote: post #15
speedgoose wrote: post #12

I prefer Ollama over the suggested alternatives. I will switch once we have good user experience on simple features. A new model is released on HF or the Ollama registry? One `ollama pull` and it's available. It's underwhelming? `ollama rm`.

> This creates a recurring pattern on r/LocalLLaMA: new model launches, people try it through Ollama, it’s broken or slow or has botched chat templates, and the model gets blamed instead of the runtime. Seems like maybe, at least some of the time, you’re being underwhelmed my ollama not the model. The better performance point alone seems worth switching away

I follow the llama.cpp runtime improvements and it’s also true for this project. They may rush a bit less but you also have to wait for a few days after a model release to get a working runtime with most features.

Re: The local LLM ecosystem doesn’t need Ollama

#28

Post by AndroTux »

0xbadcafebee wrote: post #13

No mention of the fact that Ollama is about 1000x easier to use. Llama.cpp is a great project, but it's also one of the least user friendly pieces of software I've used. I don't think anyone in the project cares about normal users. I started with Ollama, and it was great. But I moved to llama.cpp to have more up-to-date fixes. I still use Ollama to pull and list my models because it's so easy. I then built my own set…

Exactly. The blog post states that the alternatives listed are similarly intuitive. They are not. If you just need a chat app, then sure, there’s plenty of options. But if you want an OpenAI compatible API with model management, accessibility breaks down fast.

I’m open to suggestions, but the alternatives outlined in the blog post ain’t it.

Re: The local LLM ecosystem doesn’t need Ollama

#30

Post by hacker_homie »

tyfon wrote: post #6

I think the biggest advantage for me with ollama is the ability to "hotswap" models with different utility instead of restarting the server with different models combined with the simple "ollama pull model". In other words, it has been quite convenient. Due to this post I had to search a bit and it seems that llama.cpp recently got router support[1], so I need to have a look at this. My main use for this is a discord…

Llama.cpp added the ability load/switch models on demand with the max-models and models preset flags.
Post reply on HN