Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Phi-3 Technical Report

arxiv.org

Post reply on HN
Print view View original

41–50 of 132 posts

Re: Phi-3 Technical Report

#41

Post by refulgentis »

modeless wrote: post #36

Everyone needs to take these benchmark numbers with a big grain of salt. According to what I've read, Phi-2 was much worse than its benchmark numbers suggested. This model follows the same training strategy. Nobody should be assuming these numbers will translate directly into a high ranking on the LMSYS leaderboard, or usefulness in everyday tasks. Let's not dethrone Llama 3 until some real world testing can be done.…

Phi-2 wasn't chat/instruct tuned, so it didn't act good in chat, it was a base model. But the benchmark #s were real.

Re: Phi-3 Technical Report

#42

Post by refulgentis »

Deverauxi wrote: post #31
crakenzak wrote: post #4

Earlier quoted context omitted.

Can’t wait to see some Phi-3 fine tunes! Will be testing this out locally, such a small model that I can run it without quantization. Feels incredible to be living in a time with such neck breaking innovations. What are chances we’ll have a <100B parameter GPT4/Claude Opus model in the next 5 years?

5 years? 5 years is a millennia these days. We’ll have small local models beating gpt-4/Claude opus in 2024. We already have sub 100b models trading blows with former gpt-4 models, and the future is racing toward us. All these little breakthroughs are piling up.

Absolutely not on the first one. Not even close.

Re: Phi-3 Technical Report

#43

Post by saretup »

oersted wrote: post #33
jxy wrote: post #26

Earlier quoted context omitted.

This inductive logic is way overblown. > Incredible, beat Llama 3 8B with 3.8B parameters after less than a week of release. Judging by a single benchmark? Without even trying it out with real world usage? > And on LMSYS English, Llama 3 8B is on par with GPT-4 (not GPT-4-Turbo), as well as Mistral-Large. Any potential caveat in such a leaderboard not withstanding, on that leaderboard alone, there is a huge gap betwe…

It's easy to miss: select English in the dropdown. The scores are quite different in Overall and in English for LMSYS. As I've stated in other comments, yeah... Agreed, I'm stretching it a bit. It's just that any indication of a 3.8B model being in the vicinity of GPT-4 is huge. I'm sure that when things are properly measured by third-parties it will show a more sober picture. But still, with good fine-tunes, we'll p…

Firstly, English is a highly subjective category.

Secondly, Llama 3 usually adds first sentences like ‘What a unique question!’ or ‘What an insightful thought’, which might make people like it more than the competition because of the pandering.

While Llama 3 is singular in terms of size to quality ratio, calling the 8B model close to GPT4 would be an overstretch.

Re: Phi-3 Technical Report

#45

Post by bt1a »

modeless wrote: post #36

Everyone needs to take these benchmark numbers with a big grain of salt. According to what I've read, Phi-2 was much worse than its benchmark numbers suggested. This model follows the same training strategy. Nobody should be assuming these numbers will translate directly into a high ranking on the LMSYS leaderboard, or usefulness in everyday tasks. Let's not dethrone Llama 3 until some real world testing can be done.…

This won't dethrone Llama 3, but it's equally impressive.

They mention this model's relative weakness in the TruthfulQA eval, since it's more lossy trying to pack 'knowledge' into a small model relative to problem-solving skills (which shine on MMLU)

Regardless - still a very useful thing to have offline and on the fly. Those scores are nothing to scoff at.

Given that these pipelines are likely harder harder to imitate than new architectures like Transformers, I assume there has been and will be an intense focus on synthetic data generation and cleansing. Llama 3 used 15T of tokens in its training corpus vs 4.8T in the "scaled-up" version of phi-3. If you made it to the end of this disjointed ramble I'm sorry

Re: Phi-3 Technical Report

#46

Post by ukuina »

zone411 wrote: post #34
oersted wrote: post #2

Incredible, rivals Llama 3 8B with 3.8B parameters after less than a week of release. And on LMSYS English, Llama 3 8B is on par with GPT-4 (not GPT-4-Turbo), as well as Mistral-Large. Source: https://chat.lmsys.org/?leaderboard (select English in the dropdown) So we now have an open-source LLM approximately equivalent in quality to GPT-4 that can run on phones? Kinda? Wild. (I'm sure there's a lot of nuance to it, f…

> So we now have an open-source LLM approximately equivalent in quality to GPT-4 that can run on phones? No, we don't. LMsys is just one, very flawed benchmark.

Why is LMsys flawed?

Many people treat LMsys as gospel because it's the only large-scale, up-to-date qualitative benchmark. All the numeric benchmarks seem to miss real-world applicability.

Re: Phi-3 Technical Report

#47

Post by mythz »

I'll believe it till I try it for myself, Phi-2 was the clear worst of the 20 LLMs we evaluated (was also smallest so was expected).

But it was slow for its size, generated the longest responses with the most hallucinations, as well as generating the most empty responses. It was also the model ranked with the lowest quality answers.

Re: Phi-3 Technical Report

#48

Post by irjustin »

refulgentis wrote: post #41
modeless wrote: post #36

Everyone needs to take these benchmark numbers with a big grain of salt. According to what I've read, Phi-2 was much worse than its benchmark numbers suggested. This model follows the same training strategy. Nobody should be assuming these numbers will translate directly into a high ranking on the LMSYS leaderboard, or usefulness in everyday tasks. Let's not dethrone Llama 3 until some real world testing can be done.…

Phi-2 wasn't chat/instruct tuned, so it didn't act good in chat, it was a base model. But the benchmark #s were real.

I'm pretty naive so please forgive it's a stupid question.

To me, what the parent comment is saying is that even though the benchmarks are cool, it's not super helpful to the every day person. Because if you can't chat with it very well (even for a narrow context) what utility does it have with great benchmarks?

Re: Phi-3 Technical Report

#49

Post by jph00 »

coder543 wrote: post #21
ignoramous wrote: post #7

Earlier quoted context omitted.

> Phi-3-mini 3.8b: 71.2 Per the paper, phi3-mini (which is english-only) quantised to 4bit uses 1.8gb RAM and outputs 1212 tokens/sec (correction: 12 tokens/sec) on iOS. A model on par with GPT-3.5 running on phones! (weights haven't been released, though)

> (weights haven't been released, though) Phi-1, Phi-1.5, and Phi-2 have all had their weights released, and those weights are available under the MIT License. Hopefully Microsoft will continue that trend with Phi-3. > outputs 1212 tokens/sec on iOS I think you meant "12 tokens/sec", which is still nice, just a little less exciting than a kilotoken/sec.

Weights will be realised tomorrow, according to one of the tech report authors on Twitter.

Re: Phi-3 Technical Report

#50

Post by bugglebeetle »

crakenzak wrote: post #4
oersted wrote: post #2

Incredible, rivals Llama 3 8B with 3.8B parameters after less than a week of release. And on LMSYS English, Llama 3 8B is on par with GPT-4 (not GPT-4-Turbo), as well as Mistral-Large. Source: https://chat.lmsys.org/?leaderboard (select English in the dropdown) So we now have an open-source LLM approximately equivalent in quality to GPT-4 that can run on phones? Kinda? Wild. (I'm sure there's a lot of nuance to it, f…

Can’t wait to see some Phi-3 fine tunes! Will be testing this out locally, such a small model that I can run it without quantization. Feels incredible to be living in a time with such neck breaking innovations. What are chances we’ll have a <100B parameter GPT4/Claude Opus model in the next 5 years?

We already do. It’s called LLama 3 70B Instruct.
Post reply on HN