Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Phi-3 Technical Report

arxiv.org

Post reply on HN
Print view View original

111–120 of 132 posts

Re: Phi-3 Technical Report

#111

Post by azinman2 »

m3kw9 wrote: post #99

Phi-2 was useless for practical purposes except if you want to show your friends that it can write a poem, llama3 8b was slightly better but is still same category, it’s complete trash with coding vs gpt4. Llama3 400b “iS OPen SoURce!” But no you will need to pay to access because most one can not practically afford an A100 and set it up properly. What I’m trying to say is that user experience is now as key as the mo…

I just tried to give gpt4 a scrape of a menu page and asked it to reformat it to csv. It hallucinated several parts and missed others. Llama3-70b hasn’t done that. So far it’s been more reliable. You can run a quantized version on consumer hardware or pay significantly less ($10 vs $1 in, $30 vs $1 out) on hosted platforms.

Re: Phi-3 Technical Report

#112

Post by talldayo »

seydor wrote: post #63
brcmthrowaway wrote: post #9

If I was Apple I'd be quaking in my boots. They are getting too far behind to ever catch up. Nokia in 2010 vibes.

Apple's advantage is that their devices are safeguarding people from the dangers of AI

That's a very eloquent variation on the word "censorship"

Are you next going to tell us that the CIA's access to iCloud data protects their users from terrorism too?

Re: Phi-3 Technical Report

#113

Post by woodson »

andai wrote: post #97
Grimblewald wrote: post #88

Earlier quoted context omitted.

Even llama3 has its issues. Ive been quite impressed so far but if the context gets a little long it freaks out, gets stuck repeating the same token or just fails to finish an answer. This is for the full f16 8B model, so it cant be put down to quantization. It also doesnt quite handle complex instructions as well as the benchmarks would imply should.

Supposedly LLMs (especially smaller ones) are best suited to tasks where the answer is in the text, i.e. summarization, translation, and answering questions. Asking it to answer questions on its own is much more prone to hallucination. To that end I've been using Llama 3 for summarizing transcripts of YouTube videos. It does a decent job, but... every single time (literally 100% of the time), it will hallucinate a ra…

Especially for small models I had very bad results for use in translation. Even trying all kinds of tricks didn’t help (apparently prompting in the target language helps for some). Encoder-decoder models such as FLAN-T5 or MADLAD-400 seemed far superior at equal or even smaller model size.

Re: Phi-3 Technical Report

#114

Post by ashirviskas »

refulgentis wrote: post #42
Deverauxi wrote: post #31

Earlier quoted context omitted.

5 years? 5 years is a millennia these days. We’ll have small local models beating gpt-4/Claude opus in 2024. We already have sub 100b models trading blows with former gpt-4 models, and the future is racing toward us. All these little breakthroughs are piling up.

Absolutely not on the first one. Not even close.

Why not? There's still 7 months left for breakthroughs.

Re: Phi-3 Technical Report

#115

Post by 0cf8612b2e1e »

smartmic wrote: post #70

Hm, roundabout 84 authors of one "scientific" paper. I wonder if this says something about (a) the quality of its content, (b) the path were academic (?) paper publishing goes to, (c) nothing at all, or (d), something entirely else.

You should see physics. Stuff involving the large hadron collider can be pages of authors.

It costs so little to share the credit if someone was an asset.

Re: Phi-3 Technical Report

#116

Post by kiratp »

andai wrote: post #97
Grimblewald wrote: post #88

Earlier quoted context omitted.

Even llama3 has its issues. Ive been quite impressed so far but if the context gets a little long it freaks out, gets stuck repeating the same token or just fails to finish an answer. This is for the full f16 8B model, so it cant be put down to quantization. It also doesnt quite handle complex instructions as well as the benchmarks would imply should.

Supposedly LLMs (especially smaller ones) are best suited to tasks where the answer is in the text, i.e. summarization, translation, and answering questions. Asking it to answer questions on its own is much more prone to hallucination. To that end I've been using Llama 3 for summarizing transcripts of YouTube videos. It does a decent job, but... every single time (literally 100% of the time), it will hallucinate a ra…

The technique to use is to give the model an “out” for the missing/negative case.

"{text}\n\n###\n\nPlease summarize the text above. The text is a video transcript. It may not have the names of the speakers in it. If you need to refer to an unnamed speaker, call them Speaker_1, Speaker_2 and so on."

Re: Phi-3 Technical Report

#117

Post by imjonse »

Havoc wrote: post #95

Both precious phi have been epic letdowns when I actually tried them myself so quite low confidence in this being reflective of real world. Will try it anyway though

Phi-3 is instruct tuned though so hopefully better.

Re: Phi-3 Technical Report

#119

Post by ein0p »

Tried it: as soon as you ask something outside the head of the likely training data distribution it starts hallucinating like crazy. This isn’t surprising to me as a researcher: you need the associative memories of a larger model to cover the tail with at least something. That said, it’ll likely work well at specific narrow tasks once fine tuned. Just don’t expect it to really “beat GPT-3.5” at the general chat use case

Re: Phi-3 Technical Report

#120

Post by refulgentis »

ashirviskas wrote: post #114
refulgentis wrote: post #42

Earlier quoted context omitted.

Absolutely not on the first one. Not even close.

Why not? There's still 7 months left for breakthroughs.

Small leaves wiggle room, but it's extremely unlikely trad small, UX matching is a whole different matter and needs a lot of work: Worked heavily with Llama 8B over last days, and Phi 3 today, and the Q+A benchmarks don't tell the full story. Ex. It's nigh impossible to get Llama _70_B to answer in JSON; when Phi sees RAG from search it goes off inventing new RAG material and a new question.
Post reply on HN