Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

techcommunity.microsoft.com

Post reply on HN
Print view View original

31–40 of 148 posts

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#31

Post by simonw »

Havoc wrote: post #29
simonw wrote: post #9

The most interesting thing about this is the way it was trained using synthetic data, which is described in quite a bit of detail in the technical report: https://arxiv.org/abs/2412.08905 Microsoft haven't officially released the weights yet but there are unofficial GGUFs up on Hugging Face already. I tried this one: https://huggingface.co/matteogeniaccio/phi-4/tree/main I got it working with my LLM tool like this: l…

>Microsoft haven't officially released the weights Thought it was official just not on huggingface but rather whatever azure competitor thing they're pushing?

I found their AI Foundry thing so hard to figure out I couldn't tell if they had released weights (as opposed to a way of running it via an API).

Since there are GGUFs now so someone must have released some weights somewhere.

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#32

Post by Teever »

simonw wrote: post #28
bentcorner wrote: post #27

Earlier quoted context omitted.

In general I've had poor results with LLMs generating pictures using text instructions (in my case I've tried to get them to generate pictures using plots in KQL). They work but the pictures are very very basic. I'd be interested for any LLM emitting any kind of text-to-picture instructions to get results that are beyond a kindergartner-cardboard-cutout levels of art.

That's why I use the SVG pelican riding a bicycle thing as a benchmark: it's a deliberately absurd and extremely difficult task.

I'm really glad that I see someone else doing something similar. I had the epiphany a while ago that if LLMs can interpret textual instructions to draw a picture and output the design in another textual format that this a strong indicator that they're more than just stochastic parrots.

My personal test has been "A horse eating apples next to a tree" but the deliberate absurdity of your example is a much more useful test.

Do you know if this is a recognized technique that people use to study LLMs?

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#33

Post by simonw »

ant6n wrote: post #22
Terretta wrote: post #7

Earlier quoted context omitted.

Compare performance on various Macs here as it gets updated: https://github.com/ggerganov/llama.cpp/discussions/4167 OMM, Llama 3.3 70B runs at ~7 text generation tokens per second on Macbook Pro Max 128GB, while generating GPT-4 feeling text with more in depth responses and fewer bullets. Llama 3.3 70B also doesn't fight the system prompt, it leans in. Consider e.g. LM Studio (0.3.5 or newer) for a Metal (MLX) cente…

What if you have a Macbook Air with 16GB (the bechmarks dont seem to show memory).

You could definitely run an 8B model on that, and some of those are getting very capable now.

The problem is that often you can't run anything else. I've had trouble running larger models in 64GB when I've had a bunch of Firefox and VS Code tabs open at the same time.

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#34

Post by simonw »

Teever wrote: post #32
simonw wrote: post #28

Earlier quoted context omitted.

That's why I use the SVG pelican riding a bicycle thing as a benchmark: it's a deliberately absurd and extremely difficult task.

I'm really glad that I see someone else doing something similar. I had the epiphany a while ago that if LLMs can interpret textual instructions to draw a picture and output the design in another textual format that this a strong indicator that they're more than just stochastic parrots. My personal test has been "A horse eating apples next to a tree" but the deliberate absurdity of your example is a much more useful t…

I've seen people using "draw a unicorn using tikz" https://adamkdean.co.uk/posts/gpt-unicorn-a-daily-exploratio...

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#35

Post by thot_experiment »

CuriousCosmic wrote: post #24
thot_experiment wrote: post #12

For prompt adherence it still fails on tasks that Gemma2 27b nails every time. I haven't been impressed with any of the Phi family of models. The large context is very nice, though Gemma2 plays very well with self-extend.

Yeah they mention this in the weaknesses section. > While phi-4 demonstrates relatively strong performance in answering questions and performing reasoning tasks, it is less proficient at rigorously following detailed instructions, particularly those involving specific formatting requirements.

Ah good catch, I am forever cursed in my preference for snake over camel.

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#36

Post by noodletheworld »

anon373839 wrote: post #5
xeckr wrote: post #3

Looks like it punches way above its weight(s). How far are we from running a GPT-3/GPT-4 level LLM on regular consumer hardware, like a MacBook Pro?

We’re already past that point! MacBooks can easily run models exceeding GPT-3.5, such as Llama 3.1 8B, Qwen 2.5 8B, or Gemma 2 9B. These models run at very comfortable speeds on Apple Silicon. And they are distinctly more capable and less prone to hallucination than GPT-3.5 was. Llama 3.3 70B and Qwen 2.5 72B are certainly comparable to GPT-4, and they will run on MacBook Pros with at least 64GB of RAM. However, I ha…

[dead]

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#38

Post by narag »

travisgriggs wrote: post #37

Where have I been? What is a “small” language model? Wikipedia just talks about LLMs. Is this a sort of spectrum? Are there medium language models? Or is it a more nuanced classifier?

7B vs 70B parameters... I think. The small ones fit in the memory of consumer grade cards. That's what I more or less know (waiting for my new computer to arrive this week)

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#39

Post by accrual »

simonw wrote: post #28
bentcorner wrote: post #27

Earlier quoted context omitted.

In general I've had poor results with LLMs generating pictures using text instructions (in my case I've tried to get them to generate pictures using plots in KQL). They work but the pictures are very very basic. I'd be interested for any LLM emitting any kind of text-to-picture instructions to get results that are beyond a kindergartner-cardboard-cutout levels of art.

That's why I use the SVG pelican riding a bicycle thing as a benchmark: it's a deliberately absurd and extremely difficult task.

Appreciate your rapid analysis of new models, Simon. Have any models you've tested performed well on the pelican SVG task?

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#40

Post by Havoc »

simonw wrote: post #31
Havoc wrote: post #29

Earlier quoted context omitted.

>Microsoft haven't officially released the weights Thought it was official just not on huggingface but rather whatever azure competitor thing they're pushing?

I found their AI Foundry thing so hard to figure out I couldn't tell if they had released weights (as opposed to a way of running it via an API). Since there are GGUFs now so someone must have released some weights somewhere.

Yeah the weights were on there apparently.

Planned week delay between release on their own platform and hf

But much like you I decided I can be patient / use the ggufs

Post reply on HN