Llama 3.2: Revolutionizing edge AI and vision with open, customizable models • Hacker News Board

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#41

Post by refulgentis » Wed, Sep 25, 2024, 9:55 PM UTC

Closi wrote: post #31

refulgentis wrote: post #11

Earlier quoted context omitted.

Yes (rationale: 3.1 was, would be strange to rollback.) In general, you'll do a ton of damage by constraining token generation to valid JSON - I've seen models as small as 800M handle JSON with that. It's ~impossible to train constraining into it with remotely the same reliability -- you have to erase a ton of conversational training that makes it say ex. "Sure! Here's the JSON you requested:"

What about OpenAI Structured Outputs? This seems to do exactly this.

Correct, I think so too, seemed that update must be doing exactly this. tl;dr: in the context of Llama fn calling reliability, you don't need to reach for training, in fact, you'll do it and still have the same problem.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#42

Post by zackangelo » Wed, Sep 25, 2024, 9:55 PM UTC

Closi wrote: post #31

refulgentis wrote: post #11

Earlier quoted context omitted.

Yes (rationale: 3.1 was, would be strange to rollback.) In general, you'll do a ton of damage by constraining token generation to valid JSON - I've seen models as small as 800M handle JSON with that. It's ~impossible to train constraining into it with remotely the same reliability -- you have to erase a ton of conversational training that makes it say ex. "Sure! Here's the JSON you requested:"

What about OpenAI Structured Outputs? This seems to do exactly this.

I'm building this type of functionality on top of Llama models if you're interested: https://docs.mixlayer.com/examples/json-output

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#43

Post by winddude » Wed, Sep 25, 2024, 9:55 PM UTC

ushakov wrote: post #15

resters wrote: post #7

This is great! Does anyone know if the llama models are trained to do function calling like openAI models are? And/or are there any function calling training datasets?

yes, but only the text-only models! https://www.llama.com/docs/model-cards-and-prompt-formats/ll...

the vision models can also do tool calling according to the docs, but with text-only inputs, maybe that's what you meant ~ https://www.llama.com/docs/model-cards-and-prompt-formats/ll...>

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#44

Post by aftbit » Wed, Sep 25, 2024, 9:57 PM UTC

Arubis wrote: post #29

_ink_ wrote: post #26

Earlier quoted context omitted.

Oh. That's sad indeed. What might be the reason for excluding Europe?

Glibly, Europe has the gall to even consider writing regulations without asking the regulated parties for permission.

This makes it sound like some kind of retaliation, instead of Meta attempting to comply with the very regulations you're talking about. Maybe llama3.2 would violate the existing face recognition database policies?

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#45

Post by kombine » Wed, Sep 25, 2024, 9:58 PM UTC

Are these models suitable for Code assistance - as an alternative to Cursor or Copilot?

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#46

Post by DannyBee » Wed, Sep 25, 2024, 9:59 PM UTC

Arubis wrote: post #29

_ink_ wrote: post #26

Earlier quoted context omitted.

Oh. That's sad indeed. What might be the reason for excluding Europe?

Glibly, Europe has the gall to even consider writing regulations without asking the regulated parties for permission.

Why is it that and not just cost/benefit for them?

They've decided it's not worth their time/energy to do it right now in a way that complies with regulation (or whatever)

Isn't that precisely the choice the EU wants them to make?

Either do it within the bounds of what we want, or leave us out of it?

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#47

Post by TheAceOfHearts » Wed, Sep 25, 2024, 9:59 PM UTC

paxys wrote: post #25

TheAceOfHearts wrote: post #16

Earlier quoted context omitted.

I'm not implying anything. It's just frustrating that despite being a US territory with US citizens, PR isn't allowed to use this service without any explanation.

Just because you cannot access the model doesn't mean all of Puerto Rico is blocked.

When I visit meta.ai it says:

> Meta AI isn't available yet in your country

Maybe it's just my ISP, I'll ask some friends if they can access the service.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#48

Post by a_wild_dandan » Wed, Sep 25, 2024, 9:59 PM UTC

"The Llama jumped over the ______!" (Fence? River? Wall? Synagogue?)

With 1-hot encoding, the answer is "wall", with 100% probability. Oh, you gave plausibility to "fence" too? WRONG! ENJOY MORE PENALTY, SCRUB!

I believe this unforgiving dynamic is why model distillation works well. The original teacher model had to learn via the "hot or cold" game on text answers. But when the child instead imitates the teacher's predictions, it learns semantically rich answers. That strikes me as vastly more compute-efficient. So to me, it makes sense why these Llama 3.2 edge models punch so far above their weight(s). But it still blows my mind thinking how far models have advanced from a year or two ago. Kudos to Meta for these releases.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#49

Post by btdmaster » Wed, Sep 25, 2024, 10:00 PM UTC

sva_ wrote: post #21

Curious about the multimodal model's architecture. But alas, when I try to request access > Llama 3.2 Multimodal is not available in your region. It sounds like they input the continuous output of an image encoder into a transformer, similar to transfusion[0]? Does someone know where to find more details? Edit: > Regarding the licensing terms, Llama 3.2 comes with a very similar license to Llama 3.1, with one key dif…

Full text:

https://github.com/meta-llama/llama-models/blob/main/models/...

> With respect to any multimodal models included in Llama 3.2, the rights granted under Section 1(a) of the Llama 3.2 Community License Agreement are not being granted to you if you are an individual domiciled in, or a company with a principal place of business in, the European Union. This restriction does not apply to end users of a product or service that incorporates any such multimodal models.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#50

Post by paxys » Wed, Sep 25, 2024, 10:00 PM UTC

TheAceOfHearts wrote: post #47

paxys wrote: post #25

Earlier quoted context omitted.

Just because you cannot access the model doesn't mean all of Puerto Rico is blocked.

When I visit meta.ai it says: > Meta AI isn't available yet in your country Maybe it's just my ISP, I'll ask some friends if they can access the service.

meta.ai is their AI service (similar to ChatGPT). The model source itself is hosted on llama.com.