Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

techcommunity.microsoft.com

Post reply on HN
Print view View original

131–140 of 148 posts

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#131

Post by vessenes »

noman-land wrote: post #8
Terretta wrote: post #7

Earlier quoted context omitted.

Compare performance on various Macs here as it gets updated: https://github.com/ggerganov/llama.cpp/discussions/4167 OMM, Llama 3.3 70B runs at ~7 text generation tokens per second on Macbook Pro Max 128GB, while generating GPT-4 feeling text with more in depth responses and fewer bullets. Llama 3.3 70B also doesn't fight the system prompt, it leans in. Consider e.g. LM Studio (0.3.5 or newer) for a Metal (MLX) cente…

Thank you for all the tips! I'd probably go 128GB 8TB because of masochism. Curious, what makes so many of the M4s in the red currently.

It's all memory bandwidth related -- what's slow is loading these models into memory, basically. The last die from Apple with all the channels was the M2 Ultra, and I bet that's what tops those leader boards. M4 has not had a Max or an Ultra release yet; when it does (and it seems likely it will), those will be the ones to get.

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#133

Post by int_19h »

simonw wrote: post #34
Teever wrote: post #32

Earlier quoted context omitted.

I'm really glad that I see someone else doing something similar. I had the epiphany a while ago that if LLMs can interpret textual instructions to draw a picture and output the design in another textual format that this a strong indicator that they're more than just stochastic parrots. My personal test has been "A horse eating apples next to a tree" but the deliberate absurdity of your example is a much more useful t…

I've seen people using "draw a unicorn using tikz" https://adamkdean.co.uk/posts/gpt-unicorn-a-daily-exploratio...

I did some experiments of my own after this paper, but letting GPT-4 run wild, picking its own scene. It wanted to draw a boat on a lake, and I also asked it to throw in some JS animations, so it made the sun set:

https://int19h.org/chatgpt/lakeside/index.html

One interesting thing that I found out while doing this is that if you ask GPT-4 to produce SVG suitable for use in HTML, it will often just generate base64-encoded data: URIs directly. Which do contain valid SVG inside as requested.

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#134

Post by int_19h »

PhilippGille wrote: post #74
noodletheworld wrote: post #36

Earlier quoted context omitted.

[dead]

>> Llama 3.3 70B and Qwen 2.5 72B are certainly comparable to GPT-4 > I'm skeptical; the llama 3.1 405B model is the only comparable model I've used, and it's significantly larger than the 70B models you can run locally. Every new Llama generation achieved to beat larger models of the previous generation with smaller ones. Check Kagi's LLM benchmark: https://help.kagi.com/kagi/ai/llm-benchmark.html Check the HN threa…

They still do not beat GPT-4, however.

And benchmarks are very misleading in this regard. We've seen no shortage of even 8B models claiming that they beat GPT-4 and Claude in benchmarks. Every time this happens, once you start actually using the model, it's clear that it's not actually on par.

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#135

Post by int_19h »

simonw wrote: post #17
BoorishBears wrote: post #13

Earlier quoted context omitted.

Saying these models are at GPT-4 level is setting anyone who doesn't place special value on the local aspect up for disappointment. Some people do place value on running locally, and I'm not against then for it, but realistically no 70B class model has the amount of general knowledge or understanding of nuance as any recent GPT-4 checkpoint. That being said these models are still very strong compared to what we had a…

I said GPT-4, not GPT-4o. I'm talking about a model that feels equivalent to the GPT-4 we were using in March of 2023.

I remember using GPT-4 when it first dropped to get a feeling of its capabilities, and no, I wouldn't say that llama-3.3-70b is comparable.

At the end of the day, there's only so much you can cram into any given number of parameters, regardless of what any artificial benchmark says.

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#136

Post by int_19h »

chairhairair wrote: post #119
criddell wrote: post #110

Earlier quoted context omitted.

If you graded humanity on their reasoning ability, I wonder where these models would score? I think once they get to about the 85th percentile, we could upgrade the phrase to advanced reasoning. I'm roughly equating it with the percentage of the US population with at least a master's degree.

All current LLMs openly make simple mistakes that are completely incompatible with true "reasoning" (in the sense any human would have used that term years ago). I feel like I'm taking crazy pills sometimes.

If you showed the raw output of, say, QwQ-32 to any engineer from 10 years ago, I suspect they would be astonished to hear that this doesn't count as "true reasoning".

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#137

Post by simonw »

int_19h wrote: post #135
simonw wrote: post #17

Earlier quoted context omitted.

I said GPT-4, not GPT-4o. I'm talking about a model that feels equivalent to the GPT-4 we were using in March of 2023.

I remember using GPT-4 when it first dropped to get a feeling of its capabilities, and no, I wouldn't say that llama-3.3-70b is comparable. At the end of the day, there's only so much you can cram into any given number of parameters, regardless of what any artificial benchmark says.

I envy your memory.

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#138

Post by simonw »

int_19h wrote: post #134
PhilippGille wrote: post #74

Earlier quoted context omitted.

>> Llama 3.3 70B and Qwen 2.5 72B are certainly comparable to GPT-4 > I'm skeptical; the llama 3.1 405B model is the only comparable model I've used, and it's significantly larger than the 70B models you can run locally. Every new Llama generation achieved to beat larger models of the previous generation with smaller ones. Check Kagi's LLM benchmark: https://help.kagi.com/kagi/ai/llm-benchmark.html Check the HN threa…

They still do not beat GPT-4, however. And benchmarks are very misleading in this regard. We've seen no shortage of even 8B models claiming that they beat GPT-4 and Claude in benchmarks. Every time this happens, once you start actually using the model, it's clear that it's not actually on par.

GPT-4 from March 2023, not GPT-4o from May 2024.

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#139

Post by mhh__ »

simonw wrote: post #9

The most interesting thing about this is the way it was trained using synthetic data, which is described in quite a bit of detail in the technical report: https://arxiv.org/abs/2412.08905 Microsoft haven't officially released the weights yet but there are unofficial GGUFs up on Hugging Face already. I tried this one: https://huggingface.co/matteogeniaccio/phi-4/tree/main I got it working with my LLM tool like this: l…

Along those lines (synthetic data) I would keep an eye on the chinese labs given that they are probably quite data and compute constrained, in English at least.

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#140

Post by belter »

simonw wrote: post #128
belter wrote: post #113

Earlier quoted context omitted.

Is this paper wrong? - https://arxiv.org/abs/2311.09807

It shows that if you deliberately train LLMs against their own output in a loop you get problems. That's not what synthetic data training does.

I understand and appreciate your clarification. However would it not be the case some synthetic data strategies, if misapplied, can resemble the feedback loop scenario and thus risk model collapse?
Post reply on HN