Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The local LLM ecosystem doesn’t need Ollama

sleepingrobots.com

Post reply on HN
Print view View original

91–100 of 222 posts

Re: The local LLM ecosystem doesn’t need Ollama

#91

Post by operatingthetan »

tosh wrote: post #84

This is a bit like saying stop using Ubuntu, use Debian instead. Both llama.cpp and ollama are great and focused on different things and yet complement each other (both can be true at the same time!) Ollama has great ux and also supports inference via mlx, which has better performance on apple silicon than llama.cpp I'm using llama.cpp, ollama, lm studio, mlx etc etc depending on what is most convenient for me at the…

I guess read the article before commenting?

Re: The local LLM ecosystem doesn’t need Ollama

#92

Post by yuppiepuppie »

damnitbuilds wrote: post #86
tosh wrote: post #84

This is a bit like saying stop using Ubuntu, use Debian instead. Both llama.cpp and ollama are great and focused on different things and yet complement each other (both can be true at the same time!) Ollama has great ux and also supports inference via mlx, which has better performance on apple silicon than llama.cpp I'm using llama.cpp, ollama, lm studio, mlx etc etc depending on what is most convenient for me at the…

The author points out that the Ollama people are evil. So it is more like saying "Stop using SCO Unix, use Linux instead".

Where do they use the term "evil"?

Re: The local LLM ecosystem doesn’t need Ollama

#93

Post by speedgoose »

dminik wrote: post #31
speedgoose wrote: post #24

Earlier quoted context omitted.

Does it have a model registry with an API and hot swapping or you still have to use sometime like llama swap as suggested in the article ? Or is it CLI?

You can have multiple models served now with loading/unloading with just the server binary. https://github.com/ggml-org/llama.cpp/blob/master/tools/serv...

It only lacks the automatic FIFO loading/unloading then. Maybe it will be there in a few weeks.

Re: The local LLM ecosystem doesn’t need Ollama

#97

Post by cadamsdotcom »

yuppiepuppie wrote: post #92
damnitbuilds wrote: post #86

Earlier quoted context omitted.

The author points out that the Ollama people are evil. So it is more like saying "Stop using SCO Unix, use Linux instead".

Where do they use the term "evil"?

In the gaps between the tops of the lines and the bottoms of the other lines ;)

Re: The local LLM ecosystem doesn’t need Ollama

#98

Post by saghul »

em-bee wrote: post #82
Maxious wrote: post #41

Earlier quoted context omitted.

They spent several years in stealth mode but the initial release was llama.cpp. Ollama v0.0.1 "Fast inference server written in Go, powered by llama.cpp" https://github.com/ollama/ollama/tree/v0.0.1

They spent several years in stealth mode doing what? trying to build themselves what llama.cpp ended up doing for them?

I asked myself the same question. Some other commenter mentioned above they started with some Kubernetes infrastructure thing and they pivoted later.

Re: The local LLM ecosystem doesn’t need Ollama

#100

Post by samus »

cientifico wrote: post #25

For most users that wanted to run LLM locally, ollama solved the UX problem. One command, and you are running the models even with the rocm drivers without knowing. If llama provides such UX, they failed terrible at communicating that. Starting with the name. Llama.cpp: that's a cpp library! Ollama is the wrapper. That's the mental model. I don't want to build my own program! I just want to have fun :-P

How about kobold.cpp then? Or LMStudio (I know it's not open source, but at least they give proper credit to llama.cpp)?

Re curation: they should strive to not integrate broken support for models and avoid uploading broken GGUFs.

Post reply on HN