Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

github.com

Post reply on HN
Print view View original

31–40 of 65 posts

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#31

Post by refulgentis »

ynniv wrote: post #4
yjftsjthsd-h wrote: post #3

[Why] do models require a new version? It can already take arbitrary gguf; I assumed they just had a registry online

They do, and I was using the "new" models before the update. Perhaps there is tuning or bug fixes for them? Or they just want to confirm that these are supported. There are some new models that do have different architectures, so sometimes an update is necessary.

Phi 3 has a unique architecture that needed some additions to llama.cpp's conversion script. Also Phi 3 is an absolute mess, there's no reliable way to latch on to when it's done writing a message and no one wants to admit it, people are patching around it instead.

ex. I could condition on "\n\n||||", but it'd still be wrong.

Pretty much everything Phi 3 feels like it needed to all come out within 48 hours a month too early. The ONNX genai library doesn't work on Mac, at all, the mobile SDKs don't support it...sigh

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#32

Post by typpo »

anotherpaulg wrote: post #13

I actually just benchmarked Llama3 70B coding with aider, and it did quite well. It scored similar to GPT 3.5. You can use Llama3 70B with aider via Ollama [0]. It's also available for free via Groq [1] (with rate limits). And OpenRouter has it available [2] for low cost on their paid api. [0] https://aider.chat/docs/llms.html#ollama [1] https://aider.chat/docs/llms.html#groq [2] https://aider.chat/docs/llms.html#ope…

Paul's benchmarks are excellent and they're the first thing I look for to get a sense of a new model performance :)

For those looking to create their own benchmarks, promptfoo[0] is one way to do this locally:

  prompts:
    - "Write this in Python 3: {{ask}}"
  
  providers:
    - ollama:chat:llama3:8b
    - ollama:chat:phi3
    - ollama:chat:qwen:7b
    
  tests:
    - vars:
        ask: a function to determine if a number is prime
    - vars:
        ask: a function to split a restaurant bill given individual contributions and shared items
Jumping in because I'm a big believer in (1) local LLMs, and (2) evals specific to individual use cases.

[0] https://github.com/typpo/promptfoo

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#33

Post by sdesol »

refulgentis wrote: post #30
sdesol wrote: post #14

Earlier quoted context omitted.

I don't think they will move away from llama.cpp until they are forced to. The number of people contributing to llama.cpp is quite significant [1] and it wouldn't make sense to use another backend given how quickly llama.cpp is iterating and growing. [1] https://devboard.gitsense.com/ggerganov?r=ggerganov%2Fllama.... Full disclosure: This is my tool

ghost of christmas future The chance onnx becomes significantly relevant here went from 1% to 15% this week. They're demo'ing ~2x faster inference with Phi-3. There's been fits and starts on LLMs in ONNX for a year, but, with Wintel's AI PC™ push, and all the constituent parts in place (4 bit quants! adaptive quants!), I'd put very good money on it.

So you are saying Ollama is a strong MS acquisition in the future if onnx works out.

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#34

Post by tmostak »

p1esk wrote: post #25
objektif wrote: post #23

Earlier quoted context omitted.

Well being able to run these models on CPU was pretty much the revolutionary part of llama.cpp.

I can run them on CPU - HF uses plain Pytorch code - fully supported on CPU.

But it's likely to be much slower than what you'd get with a backend like llama.cpp on CPU (particularly if you're running on a Mac, but I think on Linux as well), as well as not supporting features like CPU offloading.

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#35

Post by p1esk »

tmostak wrote: post #34
p1esk wrote: post #25

Earlier quoted context omitted.

I can run them on CPU - HF uses plain Pytorch code - fully supported on CPU.

But it's likely to be much slower than what you'd get with a backend like llama.cpp on CPU (particularly if you're running on a Mac, but I think on Linux as well), as well as not supporting features like CPU offloading.

Are there benchmarks? 2x speed up would not be enough for me to return to c++ hell, but 5x might be, in some circumstances.

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#36

Post by lolinder »

p1esk wrote: post #15
wkat4242 wrote: post #11

Earlier quoted context omitted.

Hmm I don't think so. Most comments are pretty positive. I think the articles are just not really upvoted unless it's really big news, makes sense because HN is for more than just AI. But I don't think it's anti-AI like most people here would be pretty anti-cryptocurrency (and for good reason IMO)

I didn’t upvote it because I don’t use Ollama. To experiment with LLMs I use Huggingface. Does Ollama provide something I cannot get with Huggingface?

Ollama provides a web server with API that just works out of the box, which is great when you want to integrate multiple applications (potentially distributed on smaller edge devices) with LLMs that run on a single beefy machine.

In my home I have a large gaming rig that sometimes runs Ollama+Open WebUI, then I also have a bunch of other services running on a smaller server and a Raspberry Pi which reach out to Ollama for their LLM inference needs.

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#37

Post by p1esk »

lolinder wrote: post #36
p1esk wrote: post #15

Earlier quoted context omitted.

I didn’t upvote it because I don’t use Ollama. To experiment with LLMs I use Huggingface. Does Ollama provide something I cannot get with Huggingface?

Ollama provides a web server with API that just works out of the box, which is great when you want to integrate multiple applications (potentially distributed on smaller edge devices) with LLMs that run on a single beefy machine. In my home I have a large gaming rig that sometimes runs Ollama+Open WebUI, then I also have a bunch of other services running on a smaller server and a Raspberry Pi which reach out to Ollam…

Sure, maybe it’s better for niche use cases like yours.

HF is the biggest provider of llms, and I guess I haven’t run into it’s limitations yet.

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#38

Post by refulgentis »

sdesol wrote: post #33
refulgentis wrote: post #30

Earlier quoted context omitted.

ghost of christmas future The chance onnx becomes significantly relevant here went from 1% to 15% this week. They're demo'ing ~2x faster inference with Phi-3. There's been fits and starts on LLMs in ONNX for a year, but, with Wintel's AI PC™ push, and all the constituent parts in place (4 bit quants! adaptive quants!), I'd put very good money on it.

So you are saying Ollama is a strong MS acquisition in the future if onnx works out.

no, ONNX is a Microsoft project, I don't know why people know what Ollama is and I don't think they will in a year

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#39

Post by sdesol »

refulgentis wrote: post #38
sdesol wrote: post #33

Earlier quoted context omitted.

So you are saying Ollama is a strong MS acquisition in the future if onnx works out.

no, ONNX is a Microsoft project, I don't know why people know what Ollama is and I don't think they will in a year

I know it is a Microsoft Project. My reasoning is, if Ollama supports ONNX and if it can provide performance on par or better than llama.cpp, it would make sense for Microsoft to acquire Ollama for distribution reasons.

Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B

#40

Post by smcleod »

jmorgan wrote: post #24
jerrygenser wrote: post #10

I wonder if Ollama will or plans to have other "Supported backends" than llama.cpp. It's listed on the very last line of their readme as if the llama.cpp dependency is just incidental and a very minor detail rather than Ollama as a deployment mechanism for llama.cpp and gguf based models.

Yes, we are also looking at integrating MLX [1] which is optimized for Apple Silicon and built by an amazing team of individuals, a few of which were behind the original Torch [2] project. There's also TensorRT-LLM [3] by Nvidia optimized for their recent hardware. All of this of course acknowledging that llama.cpp is an incredible project with competitive performance and support for almost any platform. [1] https://…

MLX and TensorRT would be really nice!
Post reply on HN