Phi-3 Technical Report
arxiv.org
Phi-3 Technical Report
1–10 of 132 posts
Re: Phi-3 Technical Report
#2Incredible, rivals Llama 3 8B with 3.8B parameters after less than a week of release.
And on LMSYS English, Llama 3 8B is on par with GPT-4 (not GPT-4-Turbo), as well as Mistral-Large.
Source: https://chat.lmsys.org/?leaderboard (select English in the dropdown)
So we now have an open-source LLM approximately equivalent in quality to GPT-4 that can run on phones? Kinda? Wild.
(I'm sure there's a lot of nuance to it, for one these benchmarks are not so hard to game, we'll see how the dust settles, but still...)
Phi-3-mini 3.8b: 71.2
Phi-3-small 7b: 74.9
Phi-3-medium 14b: 78.2
Phi-2 2.7b: 58.8
Mistral 7b: 61.0
Gemma 7b: 62.0
Llama-3-In 8b: 68.0
Mixtral 8x7b: 69.9
GPT-3.5 1106: 75.3
(these are averages across all tasks for each model, but looking at individual scores shows a similar picture)
Re: Phi-3 Technical Report
#3I'm getting a bit skeptical of MMLU at this point. As far as I can tell it's a set of multiple choice questions that hasn't been updated since 2020. We have to trust the model providers not to deliberately or accidentally train on it for those scores to be useful.
Re: Phi-3 Technical Report
#4Incredible, rivals Llama 3 8B with 3.8B parameters after less than a week of release. And on LMSYS English, Llama 3 8B is on par with GPT-4 (not GPT-4-Turbo), as well as Mistral-Large. Source: https://chat.lmsys.org/?leaderboard (select English in the dropdown) So we now have an open-source LLM approximately equivalent in quality to GPT-4 that can run on phones? Kinda? Wild. (I'm sure there's a lot of nuance to it, f…
Can’t wait to see some Phi-3 fine tunes! Will be testing this out locally, such a small model that I can run it without quantization.
Feels incredible to be living in a time with such neck breaking innovations. What are chances we’ll have a <100B parameter GPT4/Claude Opus model in the next 5 years?
Re: Phi-3 Technical Report
#5Incredible, rivals Llama 3 8B with 3.8B parameters after less than a week of release. And on LMSYS English, Llama 3 8B is on par with GPT-4 (not GPT-4-Turbo), as well as Mistral-Large. Source: https://chat.lmsys.org/?leaderboard (select English in the dropdown) So we now have an open-source LLM approximately equivalent in quality to GPT-4 that can run on phones? Kinda? Wild. (I'm sure there's a lot of nuance to it, f…
>And on LMSYS English, Llama 3 8B is well above GPT-4
Source?
Re: Phi-3 Technical Report
#6Incredible, rivals Llama 3 8B with 3.8B parameters after less than a week of release. And on LMSYS English, Llama 3 8B is on par with GPT-4 (not GPT-4-Turbo), as well as Mistral-Large. Source: https://chat.lmsys.org/?leaderboard (select English in the dropdown) So we now have an open-source LLM approximately equivalent in quality to GPT-4 that can run on phones? Kinda? Wild. (I'm sure there's a lot of nuance to it, f…
>And on LMSYS English, Llama 3 8B is well above GPT-4 Source?
Right thanks for the reminder, I added it
Re: Phi-3 Technical Report
#7Incredible, rivals Llama 3 8B with 3.8B parameters after less than a week of release. And on LMSYS English, Llama 3 8B is on par with GPT-4 (not GPT-4-Turbo), as well as Mistral-Large. Source: https://chat.lmsys.org/?leaderboard (select English in the dropdown) So we now have an open-source LLM approximately equivalent in quality to GPT-4 that can run on phones? Kinda? Wild. (I'm sure there's a lot of nuance to it, f…
> Phi-3-mini 3.8b: 71.2
Per the paper, phi3-mini (which is english-only) quantised to 4bit uses 1.8gb RAM and outputs 1212 tokens/sec (correction: 12 tokens/sec) on iOS.
A model on par with GPT-3.5 running on phones!
(weights haven't been released, though)
Re: Phi-3 Technical Report
#8Earlier quoted context omitted.
>And on LMSYS English, Llama 3 8B is well above GPT-4 Source?
Right thanks for the reminder, I added it
Thanks, I don't see them being "well above GPT-4", merely 1 point? Also, no idea why one would want to exclude GPT-4-Turbo, the flagship "GPT-4" model, but w/e.
I also don't think they "beat Llama 3 8B"; their own abstract says "rivals that of models such as Mixtral 8x7B and GPT-3.5", "rivals" not even "beats".
Great model, but let's not overplay it.
Re: Phi-3 Technical Report
#9If I was Apple I'd be quaking in my boots. They are getting too far behind to ever catch up. Nokia in 2010 vibes.
Re: Phi-3 Technical Report
#10Less tokens than Llama 3 (3.3T vs 15T) yet better outcome. No doubt more information dense training data. The interesting thing is the use of synthetic data which they don't talk about.