Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

ai.meta.com

Post reply on HN
Print view View original

131–140 of 343 posts

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#131

Post by wrsh07 »

simonw wrote: post #128
n8henrie wrote: post #105

Earlier quoted context omitted.

I've only used ollama over cli. As per the parent poster -- do you know if there are advantages over ollama for CLI use? Have you used both?

Ollama can’t talk to OpenAI / Anthropic / etc. LLM gives you a single interface that can talk to both hosted and local models. It also logs everything you do to a SQLite database, which is great for further analysis. I use LLM and Ollama together quite a bit, because Ollama are really good at getting new models working and their server keeps those models in memory between requests.

You can run llamafile as a server, too, right? Still need to download gguf files if you don't use one of their premade binaries, but if you haven't set up llm to hit the running llamafile server I'm sure that's easy to do

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#132

Post by ravetcofx »

wongarsu wrote: post #109
tomComb wrote: post #51

Earlier quoted context omitted.

I question whether a 3B model can have “a lot of knowledge”.

From quizzing it a bit it has good knowledge but limited reasoning. For example it will tell you all about the life and death of Ho Chi Minh (and as far as I can verify factual and with more detail than what's in English Wikipedia), but when quizzed whether 2kg of feathers are heavier than 1kg of lead it will get it wrong. Though I wouldn't treat it as a domain expert on anything. For example when I asked about the s…

I wonder if spelling out the weight would work better. two kilogram for wider token input.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#133

Post by f38zf5vdt »

GaggiX wrote: post #119
f38zf5vdt wrote: post #106

Earlier quoted context omitted.

1. Ignore the benchmarks. I've been A/Bing 11B today with Molmo 72B [1], which itself has an ELO neck-and-neck with GPT4o, and it's even. Because everyone in open source tends to train on validation benchmarks, you really can not trust them. 2. The method of tokenization/adapter is novel and uses many fewer tokens than all comparable CLIP/SigLIP-adapter models, making it _much_ faster. Attention is O(n^2) on memory/c…

How about its performance compare to Qwen-2-72B tho?

Refer to the blog post I linked. Molmo is ahead of Qwen2 72b.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#134

Post by lowyek »

simonw wrote: post #55

I'm absolutely amazed at how capable the new 1B model is, considering it's just a 1.3GB download (for the Ollama GGUF version). I tried running a full codebase through it (since it can handle 128,000 tokens) and asking it to summarize the code - it did a surprisingly decent job, incomplete but still unbelievable for a model that tiny: https://gist.github.com/simonw/64c5f5b111fe473999144932bef42... More of my notes he…

Hi simon, is there a way to run the vision model easily on my mac locally?

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#135

Post by ac29 »

tomComb wrote: post #51
dhbradshaw wrote: post #19

Tried out 3B on ollama, asking questions in optics, bio, and rust. It's super fast with a lot of knowledge, a large context and great understanding. Really impressive model.

I question whether a 3B model can have “a lot of knowledge”.

As a point of comparison, the Llama 3.2 3B model is 6.5GB. The entirety of English wikipedia text is 19GB (as compressed with an algorithm from 1996, newer compression formats might do better).

Its not a perfect comparison and Llama does a lot more than English, but I would say 6.5GB of data can certainly contain a lot of knowledge.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#137

Post by lolinder »

refulgentis wrote: post #127
freedomben wrote: post #114

If anyone else is looking for the bigger models on ollama and wondering where they are, the Ollama blog post answered that for me. The are "coming soon" so they just aren't ready quite yet[1]. I was a little worried when I couldn't find them but sounds like we just need to be patient. [1]: https://ollama.com/blog/llama3.2

ollama is "just" llama.cpp underneath, I recommend switching to LM Studio or Jan, they don't have this issue of proprietary wrapper that obfuscates, you can just use any ol GGUF

What proprietary wrapper? Isn't Ollama entirely open source?

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#138

Post by dotnet00 »

ravetcofx wrote: post #132
wongarsu wrote: post #109

Earlier quoted context omitted.

From quizzing it a bit it has good knowledge but limited reasoning. For example it will tell you all about the life and death of Ho Chi Minh (and as far as I can verify factual and with more detail than what's in English Wikipedia), but when quizzed whether 2kg of feathers are heavier than 1kg of lead it will get it wrong. Though I wouldn't treat it as a domain expert on anything. For example when I asked about the s…

I wonder if spelling out the weight would work better. two kilogram for wider token input.

It still confidently said that the feathers were lighter than the lead. It did correct itself when I asked it to check again though.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#139

Post by simonw »

lowyek wrote: post #134
simonw wrote: post #55

I'm absolutely amazed at how capable the new 1B model is, considering it's just a 1.3GB download (for the Ollama GGUF version). I tried running a full codebase through it (since it can handle 128,000 tokens) and asking it to summarize the code - it did a surprisingly decent job, incomplete but still unbelievable for a model that tiny: https://gist.github.com/simonw/64c5f5b111fe473999144932bef42... More of my notes he…

Hi simon, is there a way to run the vision model easily on my mac locally?

Not that I’ve seen so far, but Ollama are pending a solution for that “soon”.
Post reply on HN