Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

techcommunity.microsoft.com

Post reply on HN
Print view View original

101–110 of 148 posts

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#101

Post by alganet »

bloomingkales wrote: post #30
xeckr wrote: post #3

Looks like it punches way above its weight(s). How far are we from running a GPT-3/GPT-4 level LLM on regular consumer hardware, like a MacBook Pro?

M4 Mac mini 16gb for $500. It's literally an inferencing block (small too, fits in my palm). I feel like the whole world needs one.

> inferencing block

Did you mean _external gpu_?

Choose any 12GB or more video card with GDDR6 or superior and you'll have at least double the performance of a base m4 mini.

The base model is almost an older generation. Thunderbolt 4 instead of 5, slower bandwidths, slower SSDs.

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#102

Post by dartos »

mirekrusin wrote: post #82
simonw wrote: post #9

The most interesting thing about this is the way it was trained using synthetic data, which is described in quite a bit of detail in the technical report: https://arxiv.org/abs/2412.08905 Microsoft haven't officially released the weights yet but there are unofficial GGUFs up on Hugging Face already. I tried this one: https://huggingface.co/matteogeniaccio/phi-4/tree/main I got it working with my LLM tool like this: l…

This "draw pelican riding on bicycle" is quite deep if you think about it. Phi is all about synthetic training and prompt -> svg -> render -> evaluate image -> feedback loop feels like ideal fit for synthetic learning. You can push it quite far with stuff like basic 2d physics etc with plotting scene after N seconds or optics/rays, magnetic force etc. SVG as LLM window to physical world.

> SVG as LLM window to physical world.

What? let’s try not to go full forehead into hype.

SVGs would be an awfully poor analogy for the physical world…

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#103

Post by michaelt »

evilduck wrote: post #56
jsheard wrote: post #19

Earlier quoted context omitted.

Yeah, a computer which starts at $3900 is really stretching that classification. Plus if you're that serious about local LLMs then you'd probably want the even bigger RAM option, which adds another $800...

An optioned up minivan is also expensive but doesn’t cost as much as a firetruck. It’s expensive but still very much consumer hardware. A 3x4090 rig is more expensive and still consumer hardware. An H100 is not, you can buy like 7 of these optioned up MBP for a single H100.

In my experience, people use the term in two separate ways.

If I'm running a software business selling software that runs on 'consumer hardware' the more people can run my software, the more people can pay me. For me, the term means the hardware used by a typical-ish consumer. I'll check the Steam hardware survey, find the 75th-percentile gamer has 8 cores, 32GB RAM, 12GB VRAM - and I'd better make sure my software works on a machine like that.

On the other hand, 'consumer hardware' could also be used to simply mean hardware available off-the-shelf from retailers who sell to consumers. By this definition, 128GB of RAM is 'consumer hardware' even if it only counts as 0.5% in Steam's hardware survey.

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#104

Post by gcanyon »

chris_st wrote: post #48
ant6n wrote: post #22

Earlier quoted context omitted.

What if you have a Macbook Air with 16GB (the bechmarks dont seem to show memory).

I have a M2 Air with 24GB, and have successfully run some 12B models such as mistral-nemo. Had other stuff going as well, but it's best to give it as much of the machine as possible.

I recently upgraded to exactly this machine for exactly this reason, but I haven't taken the leap and installed anything yet. What's your favorite model to run on it?

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#105

Post by kgwgk »

alganet wrote: post #101
bloomingkales wrote: post #30

Earlier quoted context omitted.

M4 Mac mini 16gb for $500. It's literally an inferencing block (small too, fits in my palm). I feel like the whole world needs one.

> inferencing block Did you mean _external gpu_? Choose any 12GB or more video card with GDDR6 or superior and you'll have at least double the performance of a base m4 mini. The base model is almost an older generation. Thunderbolt 4 instead of 5, slower bandwidths, slower SSDs.

> you'll have at least double the performance of a base m4 mini

For $500 all included?

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#106

Post by tosh »

jazzyjackson wrote: post #81
tosh wrote: post #68

Earlier quoted context omitted.

A Mac with 16GB RAM can run qwen 7b, gemma 9b and similar models that are somewhere between GPT3.5 and GPT4. Quite impressive.

on what metric? Why would OpenAI bother serving GPT4 if customers would be just as happy with a tiny 9B model?

https://lmarena.ai/

Check out the lmsys leaderboard. It has an overall ranking as well as ranking for specific categories.

OpenAI are also serving gpt4o mini. That said afaiu it’s not known how large/small mini is.

Being more useful than GPT3.5 is not a high bar anymore.

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#107

Post by vergessenmir »

simonw wrote: post #9

The most interesting thing about this is the way it was trained using synthetic data, which is described in quite a bit of detail in the technical report: https://arxiv.org/abs/2412.08905 Microsoft haven't officially released the weights yet but there are unofficial GGUFs up on Hugging Face already. I tried this one: https://huggingface.co/matteogeniaccio/phi-4/tree/main I got it working with my LLM tool like this: l…

When working with GGUF what chat templates do you use? Pretty much every gguf I've imported into ollama has given me garbage response. Converting the tokenizer json has yielded mixed results.

For example how do you handle the phi-4 models gguf chat template?

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#108

Post by criddell »

jsheard wrote: post #19
n144q wrote: post #15

Earlier quoted context omitted.

I wouldn't call 64GB MacBook Pro "regular consumer hardware".

Yeah, a computer which starts at $3900 is really stretching that classification. Plus if you're that serious about local LLMs then you'd probably want the even bigger RAM option, which adds another $800...

In the early 80's, people were spending more than $3k for an IBM 5150. For that price you got 64 kB of RAM, a floppy drive, and monochrome monitor.

Today, lots of people spend far more than that for gaming PCs. An Alienware R16 (unquestionably a consumer PC) with 64 GB of RAM starts at $4700.

It is an expensive computer, but the best mainstream computers at any particular time have always cost between $2500 and $5000.

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#109

Post by alganet »

kgwgk wrote: post #105
alganet wrote: post #101

Earlier quoted context omitted.

> inferencing block Did you mean _external gpu_? Choose any 12GB or more video card with GDDR6 or superior and you'll have at least double the performance of a base m4 mini. The base model is almost an older generation. Thunderbolt 4 instead of 5, slower bandwidths, slower SSDs.

> you'll have at least double the performance of a base m4 mini For $500 all included?

The base mini is 599.

Here's a config for around the same price. All brand new parts for 573. You can spend the difference improving any part you wish, or maybe get an used 3060 and go AM5 instead (Ryzen 8400F). Both paths are upgradeable.

https://pcpartpicker.com/list/ftK8rM

Double the LLM performance. Half the desktop performance. But you can use both at the same time. Your computer will not slow down when running inference.

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#110

Post by criddell »

mupuff1234 wrote: post #83

So we moved from "reasoning" to "complex reasoning". I wonder what will be next month's buzzphrase.

If you graded humanity on their reasoning ability, I wonder where these models would score?

I think once they get to about the 85th percentile, we could upgrade the phrase to advanced reasoning. I'm roughly equating it with the percentage of the US population with at least a master's degree.

Post reply on HN