Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Phi 4 available on Ollama

ollama.com

Post reply on HN
Print view View original

71–80 of 138 posts

Re: Phi 4 available on Ollama

#71

Post by FireInsight »

DrPhish wrote: post #65
gavmor wrote: post #56

Earlier quoted context omitted.

Does Ollama offer a GUI? I don't think they do. I use them because they run as a systemd service with a convenient HTTP API. That's been extremely helpful for switching between GUIs. I also like their model orgazation scheme, and the modelfile paradigm. It's also really handy that it loads and unloads models when called, which is handy for experimentation and some complex workflows eg embedding followed by inference.…

I think we agree on almost all points, but I thought ollama-gui was an official gui, so I’m even more baffled as to what the draw is. Running as a llama.cpp as a service/API endpoint is trivial (I do just that). Maybe you can outline for me what the value proposition of ollama is so I can better understand what it does that plain llama.cpp doesn’t.

Ollama allows me to use a single podman command, which uses the latest version of ollama, downloads a model of my choosing, and starts a local http endpoint widely supported by different clients. I can just run this one command to chat with a local model through a web interface, get code completions in VSCode, ask about the content of my local Markdown notes.

Now, I don't use AI that much, I could totally live without this. But if it weren't for the robust one-liner I probably wouldn't use local LLMs at all.

Re: Phi 4 available on Ollama

#72

Post by wruza »

compumetrika wrote: post #4
andhuman wrote: post #3

I’ve seen on the localllama subreddit that some GGUFs have bugs in them. The one recommended was by unsloth. However, I don’t know how the Ollama GGUF holds up.

Ollama can pull directly from HF, you just provide the URL and add to the end :Q8_0 (or whatever) to specify your desired quant. Bonus: use the short form url of `hf` instead of `huggingface` to shorten the model name a little in the ollama list table. Edit: so for example of you want the unsloth "debugged" version of Phi4, you would run: `$ollama pull hf.co/unsloth/phi-4-GGUF:Q8_0` (check on the right side of the hf…

Is it true that non-gguf models are basically all Q4-equivalent? I'm always not sure which one to download to get the "default score".

Re: Phi 4 available on Ollama

#73

Post by OhioMan2943 »

simonw wrote: post #68
OhioMan2943 wrote: post #67

Is anyone blown away by how fast we got to running something this powerful locally? I know it's easy to get burnt out on llms but this is pretty incredible. I genuinely think we're only 2 years away from full custom local voice to voice llm assistants that grow with you like JOI in BR2049 and it's going to change how we think about being human and being social, and how we grow up.

It's incredible. I've been experimenting with running local LLMs for nearly two years now, ever since the first LLaMA release back in March 2023. About six months ago I had mostly lost interest in them. They were fun to play around with but the quality difference between the ones I could run on my MacBook and the ones I could access via an online API felt insurmountable. This has completely changed in the second half…

I'm in complete agreement with your more recent timeline piece (the negative one), and as a younger user (22 year old student) I'm actively relocating this year to somewhere slightly more rural with a focus on physical/knowledge combined work to secure a good quality of life nearly solely because of how fast our timelines are.

A 'word calculator' this effective is the best substitute that we have for a logic calculator. And the fact that it's enough in 90% of situations is terrifying as it is transformative, as is the fact no one is awake to it.

Exponential power scaling in an unstable world feels like it only makes it exponentially more unstable though.

Re: Phi 4 available on Ollama

#75

Post by nothrowaways »

sgk284 wrote: post #27

Over the holidays, we published a post[1] on using high-precision few-shot examples to get `gpt-4o-mini` to perform similar to `gpt-4o`. I just re-ran that same experiment, but swapped out `gpt-4o-mini` with `phi-4`. `phi-4` really blew me away in terms of learning from few-shots. It measured as being 97% consistent with `gpt-4o` when using high-precision few-shots! Without the few-shots, it was only 37%. That's a hu…

Nice blog

Re: Phi 4 available on Ollama

#76

Post by summarity »

smallerize wrote: post #21
hbcondo714 wrote: post #10

FWIW, Phi-4 was converted to Ollama by the community last month: https://ollama.com/vanilj/Phi-4

And adopted unsloth's bug fixes a few days ago. https://ollama.com/vanilj/phi-4-unsloth

The template doesn't match Unsloth's recommendation: https://news.ycombinator.com/item?id=42662106

Re: Phi 4 available on Ollama

#77

Post by buyucu »

I have unfortunately been disappointed with the llama.cpp/ollama ecosystem of late, and thinking about moving my things to vllm instead.

llama.cpp basically dropped support for multimodal visual models. ollama still does support them, but only a handful. Also ollama still does not support vulkan eventhough llama.cpp had vulkan support for a long long time now.

This has been very sad to watch. I'm more and more convinced that vllm is the way to go, not ollama.

Re: Phi 4 available on Ollama

#78

Post by buyucu »

crorella wrote: post #9

It’s odd that MS is releasing models they are competitors to OA. This reinforce the idea that there is no real strategic advantage in owning a model. I think the strategy is now offer cheap and performant infra to run the models.

According to many press stories in the past year, the relationship between Microsoft and OpenAI has been very strained. It looks more and more like that both sides are looking for opportunity to jump ship.

This is a very clever move by Microsoft. OpenAI has no technological moat and a very unreliable partner.

Re: Phi 4 available on Ollama

#79

Post by jacooper »

buyucu wrote: post #77

I have unfortunately been disappointed with the llama.cpp/ollama ecosystem of late, and thinking about moving my things to vllm instead. llama.cpp basically dropped support for multimodal visual models. ollama still does support them, but only a handful. Also ollama still does not support vulkan eventhough llama.cpp had vulkan support for a long long time now. This has been very sad to watch. I'm more and more convin…

But can you run llms that easily with vllm? do you have to fiddle with formats to get it to run?

Re: Phi 4 available on Ollama

#80

Post by buyucu »

The_Amp_Walrus wrote: post #32
behnamoh wrote: post #11

[flagged]

I might be wrong about this but doesn't ollama do some work to ensure the model runs efficiently given your hardware? Like choosing between how much gpu memory to consume so you don't oom. Does llama.cpp do that for you with zero config?

I would even say that Ollama is a step back. For example llama.cpp supports vulkan, which is a huge gamechanger for consumer grade hardware. Ollama does not support vulkan, eventhough it's probably fairly easy to do so.

If you care about running efficiently on your hardware, then llama.cpp is they way to go, not ollama.

Post reply on HN