Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Phi 4 available on Ollama

ollama.com

Post reply on HN
Print view View original

61–70 of 138 posts

Re: Phi 4 available on Ollama

#61

Post by sgk284 »

lmeyerov wrote: post #53
sgk284 wrote: post #27

Over the holidays, we published a post[1] on using high-precision few-shot examples to get `gpt-4o-mini` to perform similar to `gpt-4o`. I just re-ran that same experiment, but swapped out `gpt-4o-mini` with `phi-4`. `phi-4` really blew me away in terms of learning from few-shots. It measured as being 97% consistent with `gpt-4o` when using high-precision few-shots! Without the few-shots, it was only 37%. That's a hu…

I like the direction, but have a pretty different experience in practice. This spans legal analytics, social media analytics, code synthesis, news analysis, cyber security LLMs, etc: 1. The only ultimate absolute quality metric I saw in that blogpost afaict was expert agreement... at 90%. All of our customers would fire us at that level across all of the diff b2b domains we work in. I'm surprised 90% is considered ac…

re: 90% – this particular case is a fairly subjective and creative task, where humans (and the LLM) are asked to follow a 22 page SOP. They've had a team of humans doing the task for 9 years, with exceptionally high variance in performance. The blended performance of the human team is meaningfully below this 90% threshold (~76%) – which speaks to the difficulty of the task.

It's, admittedly, a tough task to measure objectively though, in that it's like a code review. If a Principal Engineer pointed out 20 deficiencies in a code change and another Principal Engineer pointed out 18 of the same 20 things, but also pointed out 3 other things that the first reviewer didn't, it doesn't necessarily mean either review is wrong – they just meaningfully deviate from each other.

In this case, we chose an expert that we treat as an objective "source of truth".

re: simple tasks – We run hundreds of thousands of tasks every month with more-or-less deterministic behavior (in that, we'll reliably do it correctly a million out of a million times). We chose a particularly challenging task for the case-study though.

re: in a paying business context – FWIW, most industries are filled with humans doing tasks where the rate of perfection is far below 90%.

Re: Phi 4 available on Ollama

#62

Post by porker »

raybb wrote: post #55

I was going to ask if this or other Ollama models support structured output (like JSON). Then a quick search revealed you can as of a free weeks ago https://ollama.com/blog/structured-outputs

For structured output from anywhere I'm finding https://github.com/BoundaryML/baml good. It's more accurate than gpt-04-mini will do on its own, and any of the other JSON schema approaches I've tried.

Re: Phi 4 available on Ollama

#63

Post by jimmySixDOF »

compumetrika wrote: post #4
andhuman wrote: post #3

I’ve seen on the localllama subreddit that some GGUFs have bugs in them. The one recommended was by unsloth. However, I don’t know how the Ollama GGUF holds up.

Ollama can pull directly from HF, you just provide the URL and add to the end :Q8_0 (or whatever) to specify your desired quant. Bonus: use the short form url of `hf` instead of `huggingface` to shorten the model name a little in the ollama list table. Edit: so for example of you want the unsloth "debugged" version of Phi4, you would run: `$ollama pull hf.co/unsloth/phi-4-GGUF:Q8_0` (check on the right side of the hf…

You still need to make sure the modelfile works so this method will not run out of the box on a vision GGUF or anything with special schemas. Thats why mostly a good idea to pull from ollama directly.

Re: Phi 4 available on Ollama

#64

Post by angoragoats »

emidoots wrote: post #50
chefandy wrote: post #45

Earlier quoted context omitted.

> I have no idea what game Meta is playing Based on their business moves in recent history, I’d guess most of them are playing Farmville.

Meta's entire business model is to own users and their content. Whether it be Facebook, Instagram, Threads, Messenger, WhatsApp, etc. their focus is to acquire users, keep them in their platforms, and own their content - because /human attention is fundamentally valuable/. Meta owns 40% of the most popular social media platforms today, but their attention economies face great threats: YouTube, TikTok, Telegram, WeCha…

Can you explain how development of better generative AI (which I assume is what you mean when you say AI) will mean that “content quality across all apps/platforms can be equalized”? Unless you mean the content quality will go to shit equally everywhere (as it did in their AI profile experiment) I’m not sure I understand what you’re saying.

Re: Phi 4 available on Ollama

#65

Post by DrPhish »

gavmor wrote: post #56
DrPhish wrote: post #37

Earlier quoted context omitted.

It's very hard to put into words without coming off as being unfair to one side or the other, but the ollama project really does provide little-to-no _innovative_ value over simply running components of llama.cpp directly from the command line. 100% of the heavy lifting (from an LLM perspective) is in the llama.cpp codebase. The ollama parts are all simple, well understood, commodity components that most any develope…

Does Ollama offer a GUI? I don't think they do. I use them because they run as a systemd service with a convenient HTTP API. That's been extremely helpful for switching between GUIs. I also like their model orgazation scheme, and the modelfile paradigm. It's also really handy that it loads and unloads models when called, which is handy for experimentation and some complex workflows eg embedding followed by inference.…

I think we agree on almost all points, but I thought ollama-gui was an official gui, so I’m even more baffled as to what the draw is. Running as a llama.cpp as a service/API endpoint is trivial (I do just that). Maybe you can outline for me what the value proposition of ollama is so I can better understand what it does that plain llama.cpp doesn’t.

Re: Phi 4 available on Ollama

#66

Post by blharr »

miki123211 wrote: post #57
lolinder wrote: post #23

Earlier quoted context omitted.

That's exactly what they're saying: it's interesting that Microsoft came to the same conclusion that Meta did, that models are generally not worth keeping locked down. It suggests that OpenAI has a very fragile business model, given that they're wholly dependent on large providers for the infra, which is apparently the valuable part of the equation.

Open AI is the only company that really matters in the consumer conversational AI space. Their unique value-adds are the Chat GPT brand, being the "default destination" when people want AI, as well as all the "extra features" they add on top of raw LLMs, like the ability to do internet searches, recall facts about you from previous conversations, present data in a nice, interactive way by writing a react app, call do…

I mean they have name recognition and a userbase, but they're hardly the best at doing any of those features.

Default destination for many is still just Google, and they've added AI to their searches. AI chat boxes are shoehorned into a ton of applications and at the end of the day it'll go to the most accessible one for people. This is why AI in Windows or in your Web Browser or on your phone is a huge goal.

As far as extra features, chat GPT is a good default, but they're severely lacking compared to most other solutions out there.

Re: Phi 4 available on Ollama

#67

Post by OhioMan2943 »

Is anyone blown away by how fast we got to running something this powerful locally? I know it's easy to get burnt out on llms but this is pretty incredible.

I genuinely think we're only 2 years away from full custom local voice to voice llm assistants that grow with you like JOI in BR2049 and it's going to change how we think about being human and being social, and how we grow up.

Re: Phi 4 available on Ollama

#68

Post by simonw »

OhioMan2943 wrote: post #67

Is anyone blown away by how fast we got to running something this powerful locally? I know it's easy to get burnt out on llms but this is pretty incredible. I genuinely think we're only 2 years away from full custom local voice to voice llm assistants that grow with you like JOI in BR2049 and it's going to change how we think about being human and being social, and how we grow up.

It's incredible.

I've been experimenting with running local LLMs for nearly two years now, ever since the first LLaMA release back in March 2023.

About six months ago I had mostly lost interest in them. They were fun to play around with but the quality difference between the ones I could run on my MacBook and the ones I could access via an online API felt insurmountable.

This has completely changed in the second half of 2024. The models I can run locally had a leap in quality - they feel genuinely GPT-4 class now.

They're not as good as the best hosted models (GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet) but they're definitely good enough to be extremely useful.

This started with the Qwen 2 and 2.5 series, but I also rate Llama 3.3 70B and now Phi-4 as GPT-4 class models that run on my laptop.

I wrote more about this here: https://simonwillison.net/2024/Dec/31/llms-in-2024/#some-of-...

Re: Phi 4 available on Ollama

#69

Post by disgruntledphd2 »

DrPhish wrote: post #65
gavmor wrote: post #56

Earlier quoted context omitted.

Does Ollama offer a GUI? I don't think they do. I use them because they run as a systemd service with a convenient HTTP API. That's been extremely helpful for switching between GUIs. I also like their model orgazation scheme, and the modelfile paradigm. It's also really handy that it loads and unloads models when called, which is handy for experimentation and some complex workflows eg embedding followed by inference.…

I think we agree on almost all points, but I thought ollama-gui was an official gui, so I’m even more baffled as to what the draw is. Running as a llama.cpp as a service/API endpoint is trivial (I do just that). Maybe you can outline for me what the value proposition of ollama is so I can better understand what it does that plain llama.cpp doesn’t.

My experience of ollama is that it makes it super easy to pull various models and use them locally. Sure, I could do this myself but it's helpful not to have to.

Re: Phi 4 available on Ollama

#70

Post by timrogers »

angoragoats wrote: post #64
emidoots wrote: post #50

Earlier quoted context omitted.

Meta's entire business model is to own users and their content. Whether it be Facebook, Instagram, Threads, Messenger, WhatsApp, etc. their focus is to acquire users, keep them in their platforms, and own their content - because /human attention is fundamentally valuable/. Meta owns 40% of the most popular social media platforms today, but their attention economies face great threats: YouTube, TikTok, Telegram, WeCha…

Can you explain how development of better generative AI (which I assume is what you mean when you say AI) will mean that “content quality across all apps/platforms can be equalized”? Unless you mean the content quality will go to shit equally everywhere (as it did in their AI profile experiment) I’m not sure I understand what you’re saying.

Meta’s definition of quality is not the same as your definition of quality. For them, quality is (within reason) what drives “engagement” (aka time spent in their apps).

It might be that many people’s aesthetic sensibility is that AI-generated content is slop, but I’d still bet that tailored-perfectly-to-you content (and ads) will be highly engaging

Post reply on HN