Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

techcommunity.microsoft.com

Post reply on HN
Print view View original

91–100 of 148 posts

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#91

Post by fulafel »

belter wrote: post #89
simonw wrote: post #9

The most interesting thing about this is the way it was trained using synthetic data, which is described in quite a bit of detail in the technical report: https://arxiv.org/abs/2412.08905 Microsoft haven't officially released the weights yet but there are unofficial GGUFs up on Hugging Face already. I tried this one: https://huggingface.co/matteogeniaccio/phi-4/tree/main I got it working with my LLM tool like this: l…

> it was trained using synthetic data Is this not supposed to cause Model collapse?

No.

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#92

Post by MyFirstSass »

Teever wrote: post #32
simonw wrote: post #28

Earlier quoted context omitted.

That's why I use the SVG pelican riding a bicycle thing as a benchmark: it's a deliberately absurd and extremely difficult task.

I'm really glad that I see someone else doing something similar. I had the epiphany a while ago that if LLMs can interpret textual instructions to draw a picture and output the design in another textual format that this a strong indicator that they're more than just stochastic parrots. My personal test has been "A horse eating apples next to a tree" but the deliberate absurdity of your example is a much more useful t…

But how will that prove that it's more than a stochastic parrot, honestly curious?

Isn't it just like any kind of conversion or translation? Ie. a relationship mapping between diffrent domains and just as much parroting "known" paths between parts of different domains?

If "sun" is associated with "round", "up high", "yellow","heat" in english that will map to those things in SVG or in whatever bizarre format you throw at with relatively isomorphic paths existing there just knitted together as a different metamorphosis or cluster of nodes.

On a tangent it's interesting what constitutes the heaviest nodes in the data, how shared is "yellow" or "up high" between different domains, and what is above and below them hierarchically weight-wise. Is there a heaviest "thing in the entire dataset"?

If you dump a heatmap of a description of the sun and an SVG of a sun - of the neuron / axon like cloud of data in some model - would it look similar in some way?

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#93

Post by sabbaticaldev »

MyFirstSass wrote: post #92
Teever wrote: post #32

Earlier quoted context omitted.

I'm really glad that I see someone else doing something similar. I had the epiphany a while ago that if LLMs can interpret textual instructions to draw a picture and output the design in another textual format that this a strong indicator that they're more than just stochastic parrots. My personal test has been "A horse eating apples next to a tree" but the deliberate absurdity of your example is a much more useful t…

But how will that prove that it's more than a stochastic parrot, honestly curious? Isn't it just like any kind of conversion or translation? Ie. a relationship mapping between diffrent domains and just as much parroting "known" paths between parts of different domains? If "sun" is associated with "round", "up high", "yellow","heat" in english that will map to those things in SVG or in whatever bizarre format you thro…

that’s a huge stretch for parroting

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#94

Post by kleiba »

noman-land wrote: post #6
anon373839 wrote: post #5

Earlier quoted context omitted.

We’re already past that point! MacBooks can easily run models exceeding GPT-3.5, such as Llama 3.1 8B, Qwen 2.5 8B, or Gemma 2 9B. These models run at very comfortable speeds on Apple Silicon. And they are distinctly more capable and less prone to hallucination than GPT-3.5 was. Llama 3.3 70B and Qwen 2.5 72B are certainly comparable to GPT-4, and they will run on MacBook Pros with at least 64GB of RAM. However, I ha…

The coolness of local LLMs is THE only reason I am sadly eyeing upgrading from M1 64GB to M4/5 128+GB.

Sorry, I'm not up to date, but can you run GPTs locally or only vanilla LLMs?

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#95

Post by magicalhippo »

impossiblefork wrote: post #78
thot_experiment wrote: post #12

For prompt adherence it still fails on tasks that Gemma2 27b nails every time. I haven't been impressed with any of the Phi family of models. The large context is very nice, though Gemma2 plays very well with self-extend.

It's a much smaller model though. I think the point is more the demonstration that such a small model can have such good performance than any actual usefulness.

Gemma2 9B has significantly better prompt adherence than Llama 3.1 8B in my experience.

I've just assumed it's down to how it was trained, but no expert.

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#96

Post by rhdunn »

belter wrote: post #89
simonw wrote: post #9

The most interesting thing about this is the way it was trained using synthetic data, which is described in quite a bit of detail in the technical report: https://arxiv.org/abs/2412.08905 Microsoft haven't officially released the weights yet but there are unofficial GGUFs up on Hugging Face already. I tried this one: https://huggingface.co/matteogeniaccio/phi-4/tree/main I got it working with my LLM tool like this: l…

> it was trained using synthetic data Is this not supposed to cause Model collapse?

It depends on how you construct the synthetic data and how the model is trained on that data.

For diffusion-based image generators training only on synthetic data over repeated model training can cause model collapse as errors in the output can amplify in the trained model. It's usually the 2nd or 3rd model created this way (with output of the previous used as input for the first) for it to collapse.

It was found that using primary data along side synthetic data avoided the model collapse. Likewise, if you also have some sort of human scoring/evaluation you can help avoid artefacts.

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#97

Post by simonw »

belter wrote: post #89
simonw wrote: post #9

The most interesting thing about this is the way it was trained using synthetic data, which is described in quite a bit of detail in the technical report: https://arxiv.org/abs/2412.08905 Microsoft haven't officially released the weights yet but there are unofficial GGUFs up on Hugging Face already. I tried this one: https://huggingface.co/matteogeniaccio/phi-4/tree/main I got it working with my LLM tool like this: l…

> it was trained using synthetic data Is this not supposed to cause Model collapse?

This is why I don't think model collapse actually matters: people have been deliberately training LLMs on synthetic data for over a year at this point.

As far as I can tell model collapse happens when you deliberately train LLMs on low quality LLM-generated data so that you can write a paper about it.

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#98

Post by simonw »

codedokode wrote: post #80
simonw wrote: post #47

Earlier quoted context omitted.

gemini-exp-1206 is my new favorite: https://simonwillison.net/2024/Dec/6/gemini-exp-1206/ Claude 3.5 Sonnet is in second place: https://github.com/simonw/pelican-bicycle?tab=readme-ov-file...

They probably trained it for this specific task (generating SVG images), right?

I'm hoping that nobody has deliberately trained on SVG images of pelicans riding bicycles yet.

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#99

Post by jckahn »

ActorNightly wrote: post #65
xeckr wrote: post #3

Looks like it punches way above its weight(s). How far are we from running a GPT-3/GPT-4 level LLM on regular consumer hardware, like a MacBook Pro?

Why would you want to though? You already can get free access to large LLMs and nobody is doing anything groundbreaking with them.

I only use local, open source LLMs because I don’t trust cloud-based LLM hosts with my data. I also don’t want to build a dependence on proprietary technology.

Re: Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning

#100

Post by patrick0d »

simonw wrote: post #9

The most interesting thing about this is the way it was trained using synthetic data, which is described in quite a bit of detail in the technical report: https://arxiv.org/abs/2412.08905 Microsoft haven't officially released the weights yet but there are unofficial GGUFs up on Hugging Face already. I tried this one: https://huggingface.co/matteogeniaccio/phi-4/tree/main I got it working with my LLM tool like this: l…

this vibe check is more insightful to me than the popular evals. nice job!
Post reply on HN