Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

ai.meta.com

Post reply on HN
Print view View original

11–20 of 343 posts

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#11

Post by refulgentis »

resters wrote: post #7

This is great! Does anyone know if the llama models are trained to do function calling like openAI models are? And/or are there any function calling training datasets?

Yes (rationale: 3.1 was, would be strange to rollback.)

In general, you'll do a ton of damage by constraining token generation to valid JSON - I've seen models as small as 800M handle JSON with that. It's ~impossible to train constraining into it with remotely the same reliability -- you have to erase a ton of conversational training that makes it say ex. "Sure! Here's the JSON you requested:"

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#12

Post by TmpstsTrrctta »

resters wrote: post #7

This is great! Does anyone know if the llama models are trained to do function calling like openAI models are? And/or are there any function calling training datasets?

They mention tool calling in the link for the smaller models, and compare to 8B levels of function calling in benchmarks here:

https://news.ycombinator.com/item?id=41651126

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#13

Post by moffkalast »

I've just tested the 1B and 3B at Q8, some interesting bits:

- The 1B is extremely coherent (feels something like maybe Mistral 7B at 4 bits), and with flash attention and 4 bit KV cache it only uses about 4.2 GB of VRAM for 128k context

- A Pi 5 runs the 1B at 8.4 tok/s, haven't tested the 3B yet but it might need a lower quant to fit it and with 9T training tokens it'll probably degrade pretty badly

- The 3B is a certified Gemma-2-2B killer

Given that llama.cpp doesn't support any multimodality (they removed the old implementation), it might be a while before the 11B and 90B become runnable. Doesn't seem like they outperform Qwen-2-VL at vision benchmarks though.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#15

Post by ushakov »

resters wrote: post #7

This is great! Does anyone know if the llama models are trained to do function calling like openAI models are? And/or are there any function calling training datasets?

yes, but only the text-only models!

https://www.llama.com/docs/model-cards-and-prompt-formats/ll...

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#16

Post by TheAceOfHearts »

paxys wrote: post #9
TheAceOfHearts wrote: post #3

I still can't access the hosted model at meta.ai from Puerto Rico, despite us being U.S. citizens. I don't know what Meta has against us. Could someone try giving the 90b model this word search problem [0] and tell me how it performs? So far with every model I've tried, none has ever managed to find a single word correctly. [0] https://imgur.com/i9Ps1v6

Non US citizens can access the model just fine, if that's what you are implying.

I'm not implying anything. It's just frustrating that despite being a US territory with US citizens, PR isn't allowed to use this service without any explanation.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#18

Post by minimaxir »

Off topic/meta, but the Llama 3.2 news topic received many, many HN submissions and upvotes but never made it to the front page: the fact that it's on the front page now indicates that moderators intervened to rescue it: https://news.ycombinator.com/from?site=meta.com (showdead on)

If there's an algorithmic penalty against the news for whatever reason, that may be a flaw in the HN ranking algorithm.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#20

Post by makin »

minimaxir wrote: post #18

Off topic/meta, but the Llama 3.2 news topic received many, many HN submissions and upvotes but never made it to the front page: the fact that it's on the front page now indicates that moderators intervened to rescue it: https://news.ycombinator.com/from?site=meta.com (showdead on) If there's an algorithmic penalty against the news for whatever reason, that may be a flaw in the HN ranking algorithm.

The main issue was that Meta quickly took down the first announcement, and the only remaining working submission was the information-sparse HuggingFace link. By the time the other links were back up, it was too late. Perfect opportunity for a rescue.
Post reply on HN