Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

ai.meta.com

Post reply on HN
Print view View original

101–110 of 343 posts

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#101

Post by adtac »

a_wild_dandan wrote: post #48

"The Llama jumped over the ______!" (Fence? River? Wall? Synagogue?) With 1-hot encoding, the answer is "wall", with 100% probability. Oh, you gave plausibility to "fence" too? WRONG! ENJOY MORE PENALTY, SCRUB! I believe this unforgiving dynamic is why model distillation works well. The original teacher model had to learn via the "hot or cold" game on text answers. But when the child instead imitates the teacher's pr…

>WRONG! ENJOY MORE PENALTY, SCRUB!

Is that true tho? During training, the model predicts {"wall": 0.65, "fence": 0.25, "river": 0.03}. Then backprop modifies the weights such that it produces {"wall": 0.67, "fence": 0.24, "river": 0.02} next time.

But it does that with a much richer feedback than WRONG! because we're also telling the model how much more likely "fence" is than "wall" in an indirect way. It's likely most of the neurons that supported "wall" also supported "fence", so the average neuron that supported "river" gets penalised much more than a neuron that supported "fence".

I agree that distillation is more efficient for exactly the same reason, but I think even models as old as GPT-3 use this trick to work as well as they do.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#102

Post by dannyobrien »

GaggiX wrote: post #61
simonw wrote: post #55

I'm absolutely amazed at how capable the new 1B model is, considering it's just a 1.3GB download (for the Ollama GGUF version). I tried running a full codebase through it (since it can handle 128,000 tokens) and asking it to summarize the code - it did a surprisingly decent job, incomplete but still unbelievable for a model that tiny: https://gist.github.com/simonw/64c5f5b111fe473999144932bef42... More of my notes he…

Llama 3.2 vision models don't seem that great if they have to compare them to Claude 3 Haiku or GPT4o-mini. For an open alternative I would use Qwen-2-72B model, it's smaller than the 90B and seems to perform quite better. Also Qwen2-VL-7B as an alternative to Llama-3.2-11B, smaller, better in visual benchmarks and also Apache 2.0. Molmo models: https://huggingface.co/collections/allenai/molmo-66f379e6fe3... , also s…

What interface do you use for a locally-run Qwen2-VL-7B? Inspired by Simon Willison's research[1], I have tried it out on Hugging Face[2]. Its handwriting recognition seems fantastic, but I haven't figured out how to run it locally yet.

[1] https://simonwillison.net/2024/Sep/4/qwen2-vl/ [2] https://huggingface.co/spaces/GanymedeNil/Qwen2-VL-7B

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#103

Post by _lvbh »

404mm wrote: post #94

Newbie question, what size model would be needed to have a 10x software engineer skills and no knowledge of the human kind (ie, no need to know how to make a pizza or sequence your DNA). Is there such a model?

10x relative to what? I’ve seen bad developers use AI to 10x their productivity but they still couldn’t come anywhere close to a good developer without AI (granted, this was at a hackathon on pretty advanced optimization research. Maybe there’s more impact on lower skilled tasks)

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#104

Post by wongarsu »

create-username wrote: post #79
dhbradshaw wrote: post #19

Tried out 3B on ollama, asking questions in optics, bio, and rust. It's super fast with a lot of knowledge, a large context and great understanding. Really impressive model.

Can it speak foreign languages like German, Spanish, Ancient Greek?

Yes. It can converse perfectly normal in German. However when quizzed about German idioms it hallucinates them (in fluent German). Though that's the kind of stuff even larger models often have trouble with. For example if you ask GPT 4 about jokes in German it will give you jokes that depend on word play that only works when translated to English. In normal conversation Llama seems to speak fluent German

For Ancient Greek I just asked it (in German) to translate its previous answer to Ancient Greek, and the answer looks like Greek and according to google translate is a serviceable translation. However Llama did add a cheeky "Πηγή: Google Translate" at the end (Πηγή means source). I know little about the differences between ancient and modern Greek, but it did struggle to translate modern terms like "climate change" or "Hawaii" and added them as annotations in brackets. So I'll assume it at least tried to use Ancient Greek.

However it doesn't like switching language mid-conversation. If you start a conversation in German and after a couple messages switch to English it will understand you but answer in German. Most models switch to answering in English in that situation

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#105

Post by n8henrie »

dannyobrien wrote: post #100
jackbravo wrote: post #99

Earlier quoted context omitted.

I saw that you mention https://github.com/simonw/llm/ . Hadn't seen this before. What is its purpose? And why not use ollama instead?

llm is Simon's command line front-end to a lot of the llm apis, local and cloud-based. Along with aider-chat, it's my main interface to any LLM work -- it works well with a chat model, one-off queries, and piping text or output into a llm chain. For people who live on the command line, or are just put-off by web interfaces, it's a godsend. About the only thing I need to look further abroad for is when I'm working mul…

I've only used ollama over cli. As per the parent poster -- do you know if there are advantages over ollama for CLI use? Have you used both?

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#106

Post by f38zf5vdt »

GaggiX wrote: post #61
simonw wrote: post #55

I'm absolutely amazed at how capable the new 1B model is, considering it's just a 1.3GB download (for the Ollama GGUF version). I tried running a full codebase through it (since it can handle 128,000 tokens) and asking it to summarize the code - it did a surprisingly decent job, incomplete but still unbelievable for a model that tiny: https://gist.github.com/simonw/64c5f5b111fe473999144932bef42... More of my notes he…

Llama 3.2 vision models don't seem that great if they have to compare them to Claude 3 Haiku or GPT4o-mini. For an open alternative I would use Qwen-2-72B model, it's smaller than the 90B and seems to perform quite better. Also Qwen2-VL-7B as an alternative to Llama-3.2-11B, smaller, better in visual benchmarks and also Apache 2.0. Molmo models: https://huggingface.co/collections/allenai/molmo-66f379e6fe3... , also s…

1. Ignore the benchmarks. I've been A/Bing 11B today with Molmo 72B [1], which itself has an ELO neck-and-neck with GPT4o, and it's even. Because everyone in open source tends to train on validation benchmarks, you really can not trust them.

2. The method of tokenization/adapter is novel and uses many fewer tokens than all comparable CLIP/SigLIP-adapter models, making it _much_ faster. Attention is O(n^2) on memory/compute per sequence length.

[1] https://molmo.allenai.org/blog

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#107

Post by refulgentis »

a_wild_dandan wrote: post #48

"The Llama jumped over the ______!" (Fence? River? Wall? Synagogue?) With 1-hot encoding, the answer is "wall", with 100% probability. Oh, you gave plausibility to "fence" too? WRONG! ENJOY MORE PENALTY, SCRUB! I believe this unforgiving dynamic is why model distillation works well. The original teacher model had to learn via the "hot or cold" game on text answers. But when the child instead imitates the teacher's pr…

They don't, they're playing "hide the #s" a bit. Llama 3.2 3B is definitively worse than Phi-3 from May, both on any given metric and in an hour of playing with the 2, trying to justify moving to Llama 3.2 at 3B, given I'm adding Llama 3.2 at 1B.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#108

Post by pants2 »

404mm wrote: post #94

Newbie question, what size model would be needed to have a 10x software engineer skills and no knowledge of the human kind (ie, no need to know how to make a pizza or sequence your DNA). Is there such a model?

Most code is grounded in real-world concepts somehow. Imagine an engineer at Domino's asking it to write an ordering app. Now your model needs to know what goes in to a pizza.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#109

Post by wongarsu »

tomComb wrote: post #51
dhbradshaw wrote: post #19

Tried out 3B on ollama, asking questions in optics, bio, and rust. It's super fast with a lot of knowledge, a large context and great understanding. Really impressive model.

I question whether a 3B model can have “a lot of knowledge”.

From quizzing it a bit it has good knowledge but limited reasoning. For example it will tell you all about the life and death of Ho Chi Minh (and as far as I can verify factual and with more detail than what's in English Wikipedia), but when quizzed whether 2kg of feathers are heavier than 1kg of lead it will get it wrong.

Though I wouldn't treat it as a domain expert on anything. For example when I asked about the safety advantages of Rust over Python it oversold Rust a bit and claimed Python had issues it doesn't actually have

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#110

Post by latentsea »

404mm wrote: post #94

Newbie question, what size model would be needed to have a 10x software engineer skills and no knowledge of the human kind (ie, no need to know how to make a pizza or sequence your DNA). Is there such a model?

So long as you don't mind glue in your pizza...
Post reply on HN