Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

ai.meta.com

Post reply on HN
Print view View original

81–90 of 343 posts

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#81

Post by refulgentis »

zackangelo wrote: post #42
Closi wrote: post #31

Earlier quoted context omitted.

What about OpenAI Structured Outputs? This seems to do exactly this.

I'm building this type of functionality on top of Llama models if you're interested: https://docs.mixlayer.com/examples/json-output

I'm writing a Flutter AI client app, integrates with llama.cpp. I used a PoC of llama.cpp running in WASM, I'm desperate to signal the app is agnostic to AI provider, but it was horrifically slow, ended up backing out to WebMLC.

What are you doing underneath, here? If thats secret sauce, I'm curious what you're seeing in tokens/sec on ex. a phone vs. MacBook M-series.

Or are you deploying on servers?

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#82

Post by gunalx »

TheAceOfHearts wrote: post #3

I still can't access the hosted model at meta.ai from Puerto Rico, despite us being U.S. citizens. I don't know what Meta has against us. Could someone try giving the 90b model this word search problem [0] and tell me how it performs? So far with every model I've tried, none has ever managed to find a single word correctly. [0] https://imgur.com/i9Ps1v6

Try and find where the words are in this word puzzle undefined

''' There are two words in this word puzzle: "soup" and "mix". The word "soup" is located in the top row, and the word "mix" is located in the bottom row. ''' Edit: Tried a bit more probing like asking it to find spoon or any other word. It just makes up a row and column.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#83

Post by int_19h »

swader999 wrote: post #28
getcrunk wrote: post #23

Still no 14/30b parameter models since llama 2. Seriously killing real usability for power users/diy. The 7/8B models are great for poc and moving to edge for minor use cases … but there’s a big and empty gap till 70b that most people can’t run. The tin foil hat in me is saying this is the compromise the powers that be have agreed too. Basically being “open” but practically gimped for average joe techie. Basically ar…

You don't need an F-15 to play at least, a decent sniper rifle will do. You can still practise even with a pellet gun. I'm running 70b models on my M2 max with 96 ram. Even larger models sort of work, although I haven't really put much time into anything above 70b.

With a 128Gb Mac, you can even run 405b at 1-bit quantization - it's large enough that even with the considerable quality drop that entails, it still appears to be smarter than 70b.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#85

Post by littlestymaar »

foxhop wrote: post #71
simonw wrote: post #55

I'm absolutely amazed at how capable the new 1B model is, considering it's just a 1.3GB download (for the Ollama GGUF version). I tried running a full codebase through it (since it can handle 128,000 tokens) and asking it to summarize the code - it did a surprisingly decent job, incomplete but still unbelievable for a model that tiny: https://gist.github.com/simonw/64c5f5b111fe473999144932bef42... More of my notes he…

The llama 3.0, 3.1, & 3.2 all use the TikToken tokenizer which is the open source openai tokenizer.

GP is talking about context windows, not the number of token used by the tokenizer.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#87

Post by arnaudsm »

Is there an up-to-date leaderboard with multiple LLM benchmarks?

Livebench and Lmsys are weeks behind and sometimes refuse to add some major models. And press releases like this cherry pick their benchmarks and ignore better models like qwen2.5.

If it doesn't exist I'm willing to create it

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#88

Post by cyanydeez »

lolinder wrote: post #76
foxhop wrote: post #69

Earlier quoted context omitted.

My guess is it uses the same vocabulary size as llama 3.1 which is 128,000 different tokens (words) to support many languages. Parameter count is less of an indicator of fitness than previously thought.

That doesn't address the thing they're skeptical about, which is how much knowledge can be encoded in 3B parameters. 3B models are great for text manipulation, but I've found them to be pretty bad at having a broad understanding of pragmatics or any given subject. The larger models encode a lot more than just language in those 70B+ parameters.

Ok, but what we are probably debating is knowledge versus wisdom. Like, if I know 1+1 = 2, and I know the numbers 1 through 10, my knowledge is just 11, but my wisdom is infinite in the scope of integer addition. I can find any number, given enough time.

I'm pretty sure the AI guys are well aware of which types of models they want to produce. Models that can intake knowledge and intelligently manipulate it would mean general intelligence.

Models that can intake knowledge and only produce subsets of it's training data have a use but wouldn't be general intelligence.

Post reply on HN