Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

ai.meta.com

Post reply on HN
Print view View original

331–340 of 343 posts

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#331

Post by khafra »

KeplerBoy wrote: post #237
khafra wrote: post #193

Earlier quoted context omitted.

Of course, in many contexts, it is correct to put 9.11 after 9.9--software versioning does it that way, for example.

That's why it's an interesting question and why it struggles so hard. A good answer would explain that and state both results if the context is not hundred percent clear.

I'm not sure how useful that question is in exploring capabilities--"ask the user clarifying questions if the answer is ambiguous" is more of a rlhf or fine tune thing than a base model thing.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#332

Post by regularfry »

kristianp wrote: post #320
regularfry wrote: post #184

Earlier quoted context omitted.

Qwen2.5 has a 32B release, and quantised at q5_k_m it *just about" completely fills a 4090. It's a good model, too.

Do you also need space for context on the card to get decent speed though?

Depends how much you need. Dropping to q4_k_m gives you 3GB back if that makes the difference.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#333

Post by jona-f »

Eisenstein wrote: post #318
jona-f wrote: post #315

Earlier quoted context omitted.

Thx :). I thought the 3 looked like a b but didn't think brd would make any sense. My reasoning has led me astray.

Yeah. If you realize that a large part of the llm's 'ocr' is guessing due to context (token prediction) and not actually recognizing the characters exactly, you can see that it is indeed pretty impressive because the log it is reading uses pretty unique terminology that it couldn't know from training.

I'd say as an llm it should know this kind of stuff from training, contrary to me, for whom this is out of domain data. Anyhow I don't think the AI did a great job on that line. Would require better performance for it to be useful for me. I think larger models might actually be better at this than I am, which would be very useful.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#334

Post by sgt »

kristianp wrote: post #322
sgt wrote: post #314

Earlier quoted context omitted.

Was just concerned I don't have enough RAM. I have 16GB (M2 Pro). Got amazing mem bandwidth though (800GB/s)

M2 Pro has 200GB/s

You're right.

M1 and M2 Pro: 200GB/s

M3 Max: 300GB/s

M1/M2 Max: 400GB/s

M1/M2 Ultra: 800GB/s

Seems to be the case. An Ultra .. wow. But 200GB is also still good, so not complaining.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#335

Post by Eisenstein »

jona-f wrote: post #333
Eisenstein wrote: post #318

Earlier quoted context omitted.

Yeah. If you realize that a large part of the llm's 'ocr' is guessing due to context (token prediction) and not actually recognizing the characters exactly, you can see that it is indeed pretty impressive because the log it is reading uses pretty unique terminology that it couldn't know from training.

I'd say as an llm it should know this kind of stuff from training, contrary to me, for whom this is out of domain data. Anyhow I don't think the AI did a great job on that line. Would require better performance for it to be useful for me. I think larger models might actually be better at this than I am, which would be very useful.

Be aware that a lot of this also has to do with prompting and sampler settings. For instance changing the prompt from 'write the text on the image verbatim' to something like 'this is an electronics repair log using shorthand...' and being specific about it will give the LLM context in which to make decisions about characters and words.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#336

Post by oulipo »

Tepix wrote: post #169
oulipo wrote: post #86

Can the 3B run on a M1 macbook? It seems that it hogs all the memory. The 1B runs fine

It all depends on the amount of RAM. Since you're not mentioning it, you probably have only the default 8GB? Still, the 3B should run even on an 8GB machine.

I tried and it was so slow the system became unuseable

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#337

Post by phkahler »

grahamj wrote: post #273
phkahler wrote: post #242

Earlier quoted context omitted.

>> But Google has everything to lose doing this. LLMs are a threat to their most viable revenue stream. Just to nit pick... Advertising is their revenue stream. LLMs are a threat to search, which is what they offer people in exchange for ad views/clicks.

To nit pick even more: LLMs democratize search. They’re a threat to Google because they may allow anyone to do search as well as Google. Or better, since Google is incentivized to do search in a way that benefits them wereas prevalent LLM search may bypass that. On the flip, for all the resources they’ve poured into their models all they’ve come up with is good models, not better search. So they’re not dead in the wa…

>> So they’re not dead in the water yet but everyone suspects LLMs will eat search.

I think LLMs are acting as a store of knowledge that can answer questions. To the extent search can be replaced by asking an oracle, I agree. But search requires scoring the relevance of web pages and returning relevant results to the user. I don't see LLMs evaluating web sites like that, nor do I see them keeping up to date with news and other timely information. So I see search taking a small hit but not in significant danger from LLMs.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#339

Post by fennecbutt »

phkahler wrote: post #337
grahamj wrote: post #273

Earlier quoted context omitted.

To nit pick even more: LLMs democratize search. They’re a threat to Google because they may allow anyone to do search as well as Google. Or better, since Google is incentivized to do search in a way that benefits them wereas prevalent LLM search may bypass that. On the flip, for all the resources they’ve poured into their models all they’ve come up with is good models, not better search. So they’re not dead in the wa…

>> So they’re not dead in the water yet but everyone suspects LLMs will eat search. I think LLMs are acting as a store of knowledge that can answer questions. To the extent search can be replaced by asking an oracle, I agree. But search requires scoring the relevance of web pages and returning relevant results to the user. I don't see LLMs evaluating web sites like that, nor do I see them keeping up to date with news…

You're right. LLMs can't currently index pages like a search engine in real-time, only from training data.

Even when they hit the Internet to answer a question they're still using a search engine, ie search engines will absolutely still be required going into the future.

Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models

#340

Post by Zuiii »

grahamj wrote: post #283
alanzhuly wrote: post #148

Llama3.2 3B feels a lot better than other models with same size (e.g. Gemma2, Phi3.5-mini models). For anyone looking for a simple way to test Llama3.2 3B locally with UI, Install nexa-sdk( https://github.com/NexaAI/nexa-sdk ) and type in terminal: nexa run llama3.2 --streamlit Disclaimer: I am from Nexa AI and nexa-sdk is an open-sourced. We'd love your feedback.

or grab lmstudio

For people who really care about open source, this is not.
Post reply on HN