Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

github.com

Post reply on HN
Print view View original

41–50 of 65 posts

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#41

Post by jkh1 »

anotherpaulg wrote: post #13

I actually just benchmarked Llama3 70B coding with aider, and it did quite well. It scored similar to GPT 3.5. You can use Llama3 70B with aider via Ollama [0]. It's also available for free via Groq [1] (with rate limits). And OpenRouter has it available [2] for low cost on their paid api. [0] https://aider.chat/docs/llms.html#ollama [1] https://aider.chat/docs/llms.html#groq [2] https://aider.chat/docs/llms.html#ope…

Another benchmark: https://www.biorxiv.org/content/10.1101/2024.04.19.590278v2....

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#42

Post by jkh1 »

p1esk wrote: post #15
wkat4242 wrote: post #11

Earlier quoted context omitted.

Hmm I don't think so. Most comments are pretty positive. I think the articles are just not really upvoted unless it's really big news, makes sense because HN is for more than just AI. But I don't think it's anti-AI like most people here would be pretty anti-cryptocurrency (and for good reason IMO)

I didn’t upvote it because I don’t use Ollama. To experiment with LLMs I use Huggingface. Does Ollama provide something I cannot get with Huggingface?

Running locally is sometimes necessary, e.g. you don't want to send sensitive data to any random third party server.

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#44

Post by oulipo »

wiktor-k wrote: post #43

Ollama is simply great! I was quite surprised how easy it is to integrate through their API. A simple chat using Ollama + llama3 is less than 40 lines of TypeScript: https://github.com/wiktor-k/llama-chat

Nice! Would there be a way to do that streaming, with streaming voice input too?

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#45

Post by sh79 »

jerrygenser wrote: post #10

I wonder if Ollama will or plans to have other "Supported backends" than llama.cpp. It's listed on the very last line of their readme as if the llama.cpp dependency is just incidental and a very minor detail rather than Ollama as a deployment mechanism for llama.cpp and gguf based models.

Their behaviour around llama.cpp acknowledgement is very shady. Until the very recent, there was no mention of llama.cpp in their README at all and now it's tucked away all the way down. Compare that to the originally proposed PR for example: https://github.com/ollama/ollama/pull/3700

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#46

Post by SushiHippie »

p1esk wrote: post #35
tmostak wrote: post #34

Earlier quoted context omitted.

But it's likely to be much slower than what you'd get with a backend like llama.cpp on CPU (particularly if you're running on a Mac, but I think on Linux as well), as well as not supporting features like CPU offloading.

Are there benchmarks? 2x speed up would not be enough for me to return to c++ hell, but 5x might be, in some circumstances.

I think the biggest selling point of ollama (llama.cpp) are quantizations, for a slight hit (with q8 or q4) in quality you can get a significant performance boost.

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#48

Post by margorczynski »

sh79 wrote: post #45
jerrygenser wrote: post #10

I wonder if Ollama will or plans to have other "Supported backends" than llama.cpp. It's listed on the very last line of their readme as if the llama.cpp dependency is just incidental and a very minor detail rather than Ollama as a deployment mechanism for llama.cpp and gguf based models.

Their behaviour around llama.cpp acknowledgement is very shady. Until the very recent, there was no mention of llama.cpp in their README at all and now it's tucked away all the way down. Compare that to the originally proposed PR for example: https://github.com/ollama/ollama/pull/3700

Do you know maybe what are these alternative engines they're talking about? Or is it just a way to evade the fact that at the end of the day it is just a wrapper around llama.cpp?

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#49

Post by addandsubtract »

stephen37 wrote: post #47

I love working with Ollama, I was really surprised at how easy it is to build a simple RAG system with it. For example: https://github.com/stephen37/ollama_local_rag

Nice, I've been looking out for something like this! What's Jina AI and how is it local if I need an API key for it? Also, this is the first time I'm hearing about poetry. Might be worth including in the prerequisites (unless I can just stick with pip?)

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#50

Post by wiktor-k »

oulipo wrote: post #44
wiktor-k wrote: post #43

Ollama is simply great! I was quite surprised how easy it is to integrate through their API. A simple chat using Ollama + llama3 is less than 40 lines of TypeScript: https://github.com/wiktor-k/llama-chat

Nice! Would there be a way to do that streaming, with streaming voice input too?

Streaming is not a problem (it's just a simple flag: https://github.com/wiktor-k/llama-chat/blob/main/index.ts#L2...) but I've never used voice input.

The examples show image input though: https://github.com/ollama/ollama/blob/main/docs/api.md#reque...

Maybe you can file an issue here: https://github.com/ollama/ollama/issues

Post reply on HN