Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

github.com

Post reply on HN
Print view View original

51–60 of 65 posts

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#51

Post by sumedh »

addandsubtract wrote: post #49
stephen37 wrote: post #47

I love working with Ollama, I was really surprised at how easy it is to build a simple RAG system with it. For example: https://github.com/stephen37/ollama_local_rag

Nice, I've been looking out for something like this! What's Jina AI and how is it local if I need an API key for it? Also, this is the first time I'm hearing about poetry. Might be worth including in the prerequisites (unless I can just stick with pip?)

> What's Jina AI and how is it local if I need an API key for it?

Looks like its a vector DB used for creating and looking up embeddings (vectors). LLM is the second part of RAG, the first part is having a good embedding model.

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#52

Post by bigfudge »

jerrygenser wrote: post #10

I wonder if Ollama will or plans to have other "Supported backends" than llama.cpp. It's listed on the very last line of their readme as if the llama.cpp dependency is just incidental and a very minor detail rather than Ollama as a deployment mechanism for llama.cpp and gguf based models.

Ollama is great. I actually wish they would wrap OpenAI and Azure and generally act as as a proxy for third party APIs. Having a consistent, well thought out API which isn't tied to a single provider would be really good for the community.

Edit: this would be useful because in many cases some workloads can be local, but others cannot... e.g. if you really need gpt4 for specific queries.

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#54

Post by Zambyte »

jkh1 wrote: post #42
p1esk wrote: post #15

Earlier quoted context omitted.

I didn’t upvote it because I don’t use Ollama. To experiment with LLMs I use Huggingface. Does Ollama provide something I cannot get with Huggingface?

Running locally is sometimes necessary, e.g. you don't want to send sensitive data to any random third party server.

Both Ollama and Huggingface distribute models. The latter sites have model hosting services too, but that isn't the only way to use models from there.

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#55

Post by p1esk »

SushiHippie wrote: post #46
p1esk wrote: post #35

Earlier quoted context omitted.

Are there benchmarks? 2x speed up would not be enough for me to return to c++ hell, but 5x might be, in some circumstances.

I think the biggest selling point of ollama (llama.cpp) are quantizations, for a slight hit (with q8 or q4) in quality you can get a significant performance boost.

Does ollama/llama.cpp provide low bit operations (avx or cuda kernels) to speed up inference? Or just model compression with inference still done in fp16?

My understanding is the modern quantization algorithms are typically implemented in Pytorch.

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#56

Post by stephen37 »

addandsubtract wrote: post #49
stephen37 wrote: post #47

I love working with Ollama, I was really surprised at how easy it is to build a simple RAG system with it. For example: https://github.com/stephen37/ollama_local_rag

Nice, I've been looking out for something like this! What's Jina AI and how is it local if I need an API key for it? Also, this is the first time I'm hearing about poetry. Might be worth including in the prerequisites (unless I can just stick with pip?)

Oh yes good point, it's because at first I was using a local embedding model and then decided to use JinaAI as they have good embeddings.

Jina AI is doing different things but one of them is having embeddings and I use their English/ German embeddings as in one demo I am working with German data.

You can use pip as well but yes, let me add something about Poetry in case people don't know about it :)

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#57

Post by simonw »

p1esk wrote: post #35
tmostak wrote: post #34

Earlier quoted context omitted.

But it's likely to be much slower than what you'd get with a backend like llama.cpp on CPU (particularly if you're running on a Mac, but I think on Linux as well), as well as not supporting features like CPU offloading.

Are there benchmarks? 2x speed up would not be enough for me to return to c++ hell, but 5x might be, in some circumstances.

There's a Python binding for llama.cpp which is actively maintained and has worked well for me: https://github.com/abetlen/llama-cpp-python

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#58

Post by observationist »

sdesol wrote: post #39
refulgentis wrote: post #38

Earlier quoted context omitted.

no, ONNX is a Microsoft project, I don't know why people know what Ollama is and I don't think they will in a year

I know it is a Microsoft Project. My reasoning is, if Ollama supports ONNX and if it can provide performance on par or better than llama.cpp, it would make sense for Microsoft to acquire Ollama for distribution reasons.

Llama.cpp is the valuable bit here, and Ollama is only good for end user convenience. It saves you 20 minutes of googling and futzing with the million and one llama.cpp wrappers available for every language, lets you set up things to load on startup, but if you're building something for scale or backend, neither llama.cpp or ollama are coming along for the ride. At best it'll live through a proof of concept stage, but as soon as you start caring about performance it's getting discarded.

Microsoft isn't going to pay for something that amounts to a useful setup script wrapped around an inefficient convenience library intended for people to be able to run AI on consumer hardware. There's no exploitable value proposition, whereas building their own closed source AI systems that are tightly coupled to the Windows ecosystem and favor cloud services allows them to extract maximum rent.

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#60

Post by sh79 »

margorczynski wrote: post #48
sh79 wrote: post #45

Earlier quoted context omitted.

Their behaviour around llama.cpp acknowledgement is very shady. Until the very recent, there was no mention of llama.cpp in their README at all and now it's tucked away all the way down. Compare that to the originally proposed PR for example: https://github.com/ollama/ollama/pull/3700

Do you know maybe what are these alternative engines they're talking about? Or is it just a way to evade the fact that at the end of the day it is just a wrapper around llama.cpp?

It was mentioned in another comment to the parent. There are no alternatives currently, the whole thing has been built upon llama.cpp since its inception.
Post reply on HN