Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Phi-3 Technical Report

arxiv.org

Post reply on HN
Print view View original

81–90 of 132 posts

Re: Phi-3 Technical Report

#81

Post by samus »

torginus wrote: post #58
vessenes wrote: post #30

Earlier quoted context omitted.

Actually the original Phi papers did talk about their synthetic data strategy, and it's very cool -- essentially invert high quality textbook text using GPT-4 to create prompts, where the textbooks supply the answers. There may be more undisclosed, but it remains in my mind as one of the best ideas of the last twelve months -- so smart, and interesting, and apparently, it works well.

Except everything that comes out of an LLM (like GPT4) is highly suspect (at least in my experience).

1. They need it for style and language, not necessarily for the facts

2. Since GPT-4 is seen as the very best general-purpose LLM in existence, it makes sense to emulate its performance with less resources.

3. Phi models are also trained with other high-quality data

Re: Phi-3 Technical Report

#82

Post by fauigerzigerk »

PedroBatista wrote: post #27
brcmthrowaway wrote: post #9

If I was Apple I'd be quaking in my boots. They are getting too far behind to ever catch up. Nokia in 2010 vibes.

They'll just do what they have been doing for ~20 years, they wait, pick the "winner", polish the "user experience", call it Apple magic and incorporate that into their product cycles. Some day will be the day their joke book becomes so mediocre it will not stick anymore, but I think they are safe on this one, for now..

True for hardware, but their record on software is far less convincing.

Re: Phi-3 Technical Report

#85

Post by andy99 »

IvanAchlaqullah wrote: post #64
bt1a wrote: post #45

Earlier quoted context omitted.

This won't dethrone Llama 3, but it's equally impressive. They mention this model's relative weakness in the TruthfulQA eval, since it's more lossy trying to pack 'knowledge' into a small model relative to problem-solving skills (which shine on MMLU) Regardless - still a very useful thing to have offline and on the fly. Those scores are nothing to scoff at. Given that these pipelines are likely harder harder to imita…

> TruthfulQA Wait, people still use this benchmark? I hear there's a huge flaw on it. For examples, fine-tuning the model on 4chan make it scores better on TruthfulQA. It becomes very offensive afterwards though, for obvious reasons. See GPT-4chan [1] [1] https://www.youtube.com/watch?v=efPrtcLdcdM

Not sure I understand your example? It's not an offensiveness benchmark, in fact I can imagine a model trained to be inoffensive would do worse on a truth benchmark. I wouldn't go so far as to say truthfulQA is actually testing how truthful a model is or its reasoning. But it's one of the least correlated with other benchmarks which makes it one of the most interesting. Much more so than running most other tests that are highly correlated with MMLU performance. https://twitter.com/gblazex/status/1746295870792847562

Re: Phi-3 Technical Report

#86

Post by nurumaik »

IvanAchlaqullah wrote: post #64
bt1a wrote: post #45

Earlier quoted context omitted.

This won't dethrone Llama 3, but it's equally impressive. They mention this model's relative weakness in the TruthfulQA eval, since it's more lossy trying to pack 'knowledge' into a small model relative to problem-solving skills (which shine on MMLU) Regardless - still a very useful thing to have offline and on the fly. Those scores are nothing to scoff at. Given that these pipelines are likely harder harder to imita…

> TruthfulQA Wait, people still use this benchmark? I hear there's a huge flaw on it. For examples, fine-tuning the model on 4chan make it scores better on TruthfulQA. It becomes very offensive afterwards though, for obvious reasons. See GPT-4chan [1] [1] https://www.youtube.com/watch?v=efPrtcLdcdM

>scores better

>very offensive

Any cons?

Re: Phi-3 Technical Report

#87

Post by mirekrusin »

PedroBatista wrote: post #27
brcmthrowaway wrote: post #9

If I was Apple I'd be quaking in my boots. They are getting too far behind to ever catch up. Nokia in 2010 vibes.

They'll just do what they have been doing for ~20 years, they wait, pick the "winner", polish the "user experience", call it Apple magic and incorporate that into their product cycles. Some day will be the day their joke book becomes so mediocre it will not stick anymore, but I think they are safe on this one, for now..

Considering that experiments cost tens to hundreds of millions of dollars a pop this may be not that bad strategy.

Re: Phi-3 Technical Report

#88

Post by Grimblewald »

bt1a wrote: post #45
modeless wrote: post #36

Everyone needs to take these benchmark numbers with a big grain of salt. According to what I've read, Phi-2 was much worse than its benchmark numbers suggested. This model follows the same training strategy. Nobody should be assuming these numbers will translate directly into a high ranking on the LMSYS leaderboard, or usefulness in everyday tasks. Let's not dethrone Llama 3 until some real world testing can be done.…

This won't dethrone Llama 3, but it's equally impressive. They mention this model's relative weakness in the TruthfulQA eval, since it's more lossy trying to pack 'knowledge' into a small model relative to problem-solving skills (which shine on MMLU) Regardless - still a very useful thing to have offline and on the fly. Those scores are nothing to scoff at. Given that these pipelines are likely harder harder to imita…

Even llama3 has its issues. Ive been quite impressed so far but if the context gets a little long it freaks out, gets stuck repeating the same token or just fails to finish an answer. This is for the full f16 8B model, so it cant be put down to quantization. It also doesnt quite handle complex instructions as well as the benchmarks would imply should.

Re: Phi-3 Technical Report

#89

Post by ankit219 »

modeless wrote: post #36

Everyone needs to take these benchmark numbers with a big grain of salt. According to what I've read, Phi-2 was much worse than its benchmark numbers suggested. This model follows the same training strategy. Nobody should be assuming these numbers will translate directly into a high ranking on the LMSYS leaderboard, or usefulness in everyday tasks. Let's not dethrone Llama 3 until some real world testing can be done.…

Not trying to disparage them, but their models always give a feeling that it is overfitted on benchmarks hence they perform so well. On everyday tasks, it's much worse - chat or simple completion tasks.

Distilling can work and there are papers which suggest it does, but we still do not have a reliable mechanism which can distill knowledge from larger teacher models to smaller student models.

Re: Phi-3 Technical Report

#90

Post by candiodari »

viraptor wrote: post #72
oersted wrote: post #2

Incredible, rivals Llama 3 8B with 3.8B parameters after less than a week of release. And on LMSYS English, Llama 3 8B is on par with GPT-4 (not GPT-4-Turbo), as well as Mistral-Large. Source: https://chat.lmsys.org/?leaderboard (select English in the dropdown) So we now have an open-source LLM approximately equivalent in quality to GPT-4 that can run on phones? Kinda? Wild. (I'm sure there's a lot of nuance to it, f…

On par in some categories. Phi was intended for reasoning, not storing data, due to small size. I mean, it's still great, but the smaller it gets, the more facts from outside of the prompts context will not be known at all.

I wonder if that's a positive or negative. How does it affect hallucinations?
Post reply on HN