Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

ai.meta.com

Post reply on HN
Print view View original

31–40 of 343 posts

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#31

Post by Closi »

refulgentis wrote: post #11
resters wrote: post #7

This is great! Does anyone know if the llama models are trained to do function calling like openAI models are? And/or are there any function calling training datasets?

Yes (rationale: 3.1 was, would be strange to rollback.) In general, you'll do a ton of damage by constraining token generation to valid JSON - I've seen models as small as 800M handle JSON with that. It's ~impossible to train constraining into it with remotely the same reliability -- you have to erase a ton of conversational training that makes it say ex. "Sure! Here's the JSON you requested:"

What about OpenAI Structured Outputs? This seems to do exactly this.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#32

Post by mrfinn »

sva_ wrote: post #21

Curious about the multimodal model's architecture. But alas, when I try to request access > Llama 3.2 Multimodal is not available in your region. It sounds like they input the continuous output of an image encoder into a transformer, similar to transfusion[0]? Does someone know where to find more details? Edit: > Regarding the licensing terms, Llama 3.2 comes with a very similar license to Llama 3.1, with one key dif…

Pity, it's over. We'll never ever be able to download those ten gigabytes files, at the other side of the fence.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#34

Post by Y_Y »

sva_ wrote: post #21

Curious about the multimodal model's architecture. But alas, when I try to request access > Llama 3.2 Multimodal is not available in your region. It sounds like they input the continuous output of an image encoder into a transformer, similar to transfusion[0]? Does someone know where to find more details? Edit: > Regarding the licensing terms, Llama 3.2 comes with a very similar license to Llama 3.1, with one key dif…

I hereby grant license to anyone in the EU to do whatever they want with this.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#35

Post by pocketarc »

Arubis wrote: post #29
_ink_ wrote: post #26

Earlier quoted context omitted.

Oh. That's sad indeed. What might be the reason for excluding Europe?

Glibly, Europe has the gall to even consider writing regulations without asking the regulated parties for permission.

Between this and Apple's policies, big tech corporations really seem to be putting the screws to the EU as much as they can.

"See, consumers? Look at how bad your regulation is, that you're missing out on all these cool things we're working on. Talk to your politicians!"

Regardless of your political opinion on the subject, you've got to admit, at the very least, it will be educational to see how this develops over the next 5-10 years of tech progress, as the EU gets excluded from more and more things.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#37

Post by Patrick_Devine »

moffkalast wrote: post #13

I've just tested the 1B and 3B at Q8, some interesting bits: - The 1B is extremely coherent (feels something like maybe Mistral 7B at 4 bits), and with flash attention and 4 bit KV cache it only uses about 4.2 GB of VRAM for 128k context - A Pi 5 runs the 1B at 8.4 tok/s, haven't tested the 3B yet but it might need a lower quant to fit it and with 9T training tokens it'll probably degrade pretty badly - The 3B is a c…

Hoping to get this out soon w/ Ollama. Just working out a couple of last kinks. The 11b model is legit good though, particularly for tasks like OCR. It can actually read my cursive handwriting.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#38

Post by Y_Y »

sk11001 wrote: post #33

Can one of thse models be run on a single machine? What specs do you need?

Absolutely! They have a billion-parameter model that will run on my first computer if we quantize it to 1.5 bits. But realistically yes, if you can fit in total ram you can run it slowly, if you can fit it in gpu ram you can probably run it fast enough to chat.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#40

Post by zackangelo »

ushakov wrote: post #15
resters wrote: post #7

This is great! Does anyone know if the llama models are trained to do function calling like openAI models are? And/or are there any function calling training datasets?

yes, but only the text-only models! https://www.llama.com/docs/model-cards-and-prompt-formats/ll...

This is incorrect:

> With text-only inputs, the Llama 3.2 Vision Models can do tool-calling exactly like their Llama 3.1 Text Model counterparts. You can use either the system or user prompts to provide the function definitions.

> Currently the vision models don’t support tool-calling with text+image inputs.

They support it, but not when an image is submitted in the prompt. I'd be curious to see what the model does. Meta typically sets conservative expectations around this type of behavior (e.g., they say that the 3.1 8b model won't do multiple tool calls, but in my experience it does so just fine).

Post reply on HN