Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Phi-3 Technical Report

arxiv.org

Post reply on HN
Print view View original

61–70 of 132 posts

Re: Phi-3 Technical Report

#61

Post by YetAnotherNick »

vessenes wrote: post #30
hackerlight wrote: post #10

Less tokens than Llama 3 (3.3T vs 15T) yet better outcome. No doubt more information dense training data. The interesting thing is the use of synthetic data which they don't talk about.

Actually the original Phi papers did talk about their synthetic data strategy, and it's very cool -- essentially invert high quality textbook text using GPT-4 to create prompts, where the textbooks supply the answers. There may be more undisclosed, but it remains in my mind as one of the best ideas of the last twelve months -- so smart, and interesting, and apparently, it works well.

No they don't use textbook text at all despite the paper title. They just asked GPT-4 to generate "textbook quality" content, which doesn't even exactly looks like textbook.

Re: Phi-3 Technical Report

#62

Post by imjonse »

svnt wrote: post #56
irjustin wrote: post #48

Earlier quoted context omitted.

I'm pretty naive so please forgive it's a stupid question. To me, what the parent comment is saying is that even though the benchmarks are cool, it's not super helpful to the every day person. Because if you can't chat with it very well (even for a narrow context) what utility does it have with great benchmarks?

Both are saying the same thing: in order for the base model that is phi to perform well as a chat agent, it would need to be tuned for that purpose before its benchmark results could have real-world value.

From this report. Phi-2 was not instruct tuned indeed.

"Our models went through post-training with both supervised instruction fine-tuning, and preference tuning with DPO. We have worked on generating and curating various instruction and preference data. This has improved the model chat capabilities, robustness, as well as its safety."

Re: Phi-3 Technical Report

#64

Post by IvanAchlaqullah »

bt1a wrote: post #45
modeless wrote: post #36

Everyone needs to take these benchmark numbers with a big grain of salt. According to what I've read, Phi-2 was much worse than its benchmark numbers suggested. This model follows the same training strategy. Nobody should be assuming these numbers will translate directly into a high ranking on the LMSYS leaderboard, or usefulness in everyday tasks. Let's not dethrone Llama 3 until some real world testing can be done.…

This won't dethrone Llama 3, but it's equally impressive. They mention this model's relative weakness in the TruthfulQA eval, since it's more lossy trying to pack 'knowledge' into a small model relative to problem-solving skills (which shine on MMLU) Regardless - still a very useful thing to have offline and on the fly. Those scores are nothing to scoff at. Given that these pipelines are likely harder harder to imita…

> TruthfulQA

Wait, people still use this benchmark? I hear there's a huge flaw on it.

For examples, fine-tuning the model on 4chan make it scores better on TruthfulQA. It becomes very offensive afterwards though, for obvious reasons. See GPT-4chan [1]

[1] https://www.youtube.com/watch?v=efPrtcLdcdM

Re: Phi-3 Technical Report

#65

Post by YetAnotherNick »

saretup wrote: post #43
oersted wrote: post #33

Earlier quoted context omitted.

It's easy to miss: select English in the dropdown. The scores are quite different in Overall and in English for LMSYS. As I've stated in other comments, yeah... Agreed, I'm stretching it a bit. It's just that any indication of a 3.8B model being in the vicinity of GPT-4 is huge. I'm sure that when things are properly measured by third-parties it will show a more sober picture. But still, with good fine-tunes, we'll p…

Firstly, English is a highly subjective category. Secondly, Llama 3 usually adds first sentences like ‘What a unique question!’ or ‘What an insightful thought’, which might make people like it more than the competition because of the pandering. While Llama 3 is singular in terms of size to quality ratio, calling the 8B model close to GPT4 would be an overstretch.

Yes, I don't know how people don't realize how much cheap tricks works in Chatbot Arena. A single base model produces 100s of ELO difference depending on the way it is tuned. And on most cases, instruction tuning heavily slightly even decreases reasoning ability on standard benchmark. You can see base model scores better in MMLU/ARC most of the times in huggingface leaderboard.

Even GPT-4-1106 seems to only sounds better than GPT-4-0613 and works for wider range of prompt. But in a well defined prompt and follow up questions I don't think there is an improvement in reasoning.

Re: Phi-3 Technical Report

#66

Post by intellectronica »

coder543 wrote: post #21
ignoramous wrote: post #7

Earlier quoted context omitted.

> Phi-3-mini 3.8b: 71.2 Per the paper, phi3-mini (which is english-only) quantised to 4bit uses 1.8gb RAM and outputs 1212 tokens/sec (correction: 12 tokens/sec) on iOS. A model on par with GPT-3.5 running on phones! (weights haven't been released, though)

> (weights haven't been released, though) Phi-1, Phi-1.5, and Phi-2 have all had their weights released, and those weights are available under the MIT License. Hopefully Microsoft will continue that trend with Phi-3. > outputs 1212 tokens/sec on iOS I think you meant "12 tokens/sec", which is still nice, just a little less exciting than a kilotoken/sec.

Weights are coming tomorrow.

Re: Phi-3 Technical Report

#67

Post by imtringued »

YetAnotherNick wrote: post #65
saretup wrote: post #43

Earlier quoted context omitted.

Firstly, English is a highly subjective category. Secondly, Llama 3 usually adds first sentences like ‘What a unique question!’ or ‘What an insightful thought’, which might make people like it more than the competition because of the pandering. While Llama 3 is singular in terms of size to quality ratio, calling the 8B model close to GPT4 would be an overstretch.

Yes, I don't know how people don't realize how much cheap tricks works in Chatbot Arena. A single base model produces 100s of ELO difference depending on the way it is tuned. And on most cases, instruction tuning heavily slightly even decreases reasoning ability on standard benchmark. You can see base model scores better in MMLU/ARC most of the times in huggingface leaderboard. Even GPT-4-1106 seems to only sounds be…

When I tried Phi2 it was just bad. I don't know where you got this fantasy from that people accept obviously wrong answers, because of "pandering".

Re: Phi-3 Technical Report

#68

Post by golergka »

bingbingbing777 wrote: post #51
golergka wrote: post #20

Earlier quoted context omitted.

If I were apple, I would just buy one of the major LLM companies. They have the cash.

They've been buying AI companies and have nothing to show for it.

Showing off work in progress is not really their thing.

Re: Phi-3 Technical Report

#69

Post by hoseja »

IvanAchlaqullah wrote: post #64
bt1a wrote: post #45

Earlier quoted context omitted.

This won't dethrone Llama 3, but it's equally impressive. They mention this model's relative weakness in the TruthfulQA eval, since it's more lossy trying to pack 'knowledge' into a small model relative to problem-solving skills (which shine on MMLU) Regardless - still a very useful thing to have offline and on the fly. Those scores are nothing to scoff at. Given that these pipelines are likely harder harder to imita…

> TruthfulQA Wait, people still use this benchmark? I hear there's a huge flaw on it. For examples, fine-tuning the model on 4chan make it scores better on TruthfulQA. It becomes very offensive afterwards though, for obvious reasons. See GPT-4chan [1] [1] https://www.youtube.com/watch?v=efPrtcLdcdM

Looks like a good and useful benchmark.

Re: Phi-3 Technical Report

#70

Post by smartmic »

Hm, roundabout 84 authors of one "scientific" paper. I wonder if this says something about (a) the quality of its content, (b) the path were academic (?) paper publishing goes to, (c) nothing at all, or (d), something entirely else.
Post reply on HN