[deleted]
Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
231–240 of 343 posts
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#232Earlier quoted context omitted.
I question whether a 3B model can have “a lot of knowledge”.
From quizzing it a bit it has good knowledge but limited reasoning. For example it will tell you all about the life and death of Ho Chi Minh (and as far as I can verify factual and with more detail than what's in English Wikipedia), but when quizzed whether 2kg of feathers are heavier than 1kg of lead it will get it wrong. Though I wouldn't treat it as a domain expert on anything. For example when I asked about the s…
Well the feathers heavier than lead thing is definitely somewhere in training data.
Imo we should be testing reasoning for these models by presenting things or situations that neither the human or machine has seen or experienced.
Think; how often do humans have a truly new experience with no basis on past ones? Very rarely - even learning to ride a bike it could be presumed that it has a link to walking/running and movement in general.
Even human "creativity" (much ado about nothing) is creating drama in the AI space...but I find this a super interesting topic as essentially 99.9999% of all human "creativity" is just us rehashing and borrowing heavily from stuff we've seen or encountered in nature. What are elves, dwarves, etc than people with slightly unusual features. Even aliens we create are based on: humans/bipedal, squid/sea creature, dragon/reptile, etc. How often does human creativity really, _really_ come up with something novel? Almost never!
Edit: I think my overarching point is that we need to come up with better exercises to test these models, but it's almost impossible for us to do this because most of us are incapable of creating purely novel concepts and ideas. AGI perhaps isn't that far off given that humans have been the stochastic parrots all along.
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#233Earlier quoted context omitted.
Llama 3.2 vision models don't seem that great if they have to compare them to Claude 3 Haiku or GPT4o-mini. For an open alternative I would use Qwen-2-72B model, it's smaller than the 90B and seems to perform quite better. Also Qwen2-VL-7B as an alternative to Llama-3.2-11B, smaller, better in visual benchmarks and also Apache 2.0. Molmo models: https://huggingface.co/collections/allenai/molmo-66f379e6fe3... , also s…
1. Ignore the benchmarks. I've been A/Bing 11B today with Molmo 72B [1], which itself has an ELO neck-and-neck with GPT4o, and it's even. Because everyone in open source tends to train on validation benchmarks, you really can not trust them. 2. The method of tokenization/adapter is novel and uses many fewer tokens than all comparable CLIP/SigLIP-adapter models, making it _much_ faster. Attention is O(n^2) on memory/c…
> I've been A/Bing 11B today with Molmo 72B
How are you testing Molmo 72B? If you are interacting with https://molmo.allenai.org/, they are using Molmo-7B-D.
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#234I'm absolutely amazed at how capable the new 1B model is, considering it's just a 1.3GB download (for the Ollama GGUF version). I tried running a full codebase through it (since it can handle 128,000 tokens) and asking it to summarize the code - it did a surprisingly decent job, incomplete but still unbelievable for a model that tiny: https://gist.github.com/simonw/64c5f5b111fe473999144932bef42... More of my notes he…
If you are in US, you get 1 billion tokens a DAY with Gemini (Google) completely free of cost. Gemini Flash is fast with upto 4 million token context. Gemini Flash 002 improved in math and logical abilities surpassing Claude and Gpt 4o You can simply use Gemini Flash for Code Completion, git review tool and many more.
Is this sustainable though, or are they just trying really hard to attract users? If I build all of my tooling on it, will they start charging me thousands of dollars next year once the subsidies dry up? With a local model running with open source software, at least I can know that as long as my computer can still compute, the model will still run just as well and just as fast as it did on day 1, and cost the same amount of electricity
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#235Earlier quoted context omitted.
1. Ignore the benchmarks. I've been A/Bing 11B today with Molmo 72B [1], which itself has an ELO neck-and-neck with GPT4o, and it's even. Because everyone in open source tends to train on validation benchmarks, you really can not trust them. 2. The method of tokenization/adapter is novel and uses many fewer tokens than all comparable CLIP/SigLIP-adapter models, making it _much_ faster. Attention is O(n^2) on memory/c…
I tried some OCR use cases, Claude Sonnet just blows Molmo.
When you say "blows," do you mean in a subservient sense or more like, "it blows it out of the water?"
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#236Anyone on HN running models on their own local machines, like smaller Llama models or such? Or something else?
For sure dude! Top comment thread is all about using ollama and other ways to get that done.
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#237Tried the 1B model with the "think step by step" prompt. It gets "which is larger: 9.11 or 9.9?" right if it manages to mention that decimals need to be compared first in its step-by-step thinking. If it skips mentioning decimals, then it says 9.11 is larger. It gets the strawberry question wrong even after enumerating all the letters correctly, probably because it can't properly count.
Of course, in many contexts, it is correct to put 9.11 after 9.9--software versioning does it that way, for example.
That's why it's an interesting question and why it struggles so hard.
A good answer would explain that and state both results if the context is not hundred percent clear.
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#238Earlier quoted context omitted.
If you are in US, you get 1 billion tokens a DAY with Gemini (Google) completely free of cost. Gemini Flash is fast with upto 4 million token context. Gemini Flash 002 improved in math and logical abilities surpassing Claude and Gpt 4o You can simply use Gemini Flash for Code Completion, git review tool and many more.
Is this sustainable though, or are they just trying really hard to attract users? If I build all of my tooling on it, will they start charging me thousands of dollars next year once the subsidies dry up? With a local model running with open source software, at least I can know that as long as my computer can still compute, the model will still run just as well and just as fast as it did on day 1, and cost the same am…
Are you asking whether giving away $5/day/user (what OpenAI charges) in compute is sustainable?
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#239Earlier quoted context omitted.
Naah, Qwen2-VL-7b still is much much better than 11b model for handwritten OCR from what i have tested. The 11b model hallucinates in case of handwritten OCR.
Where can I try it out. The playground on their homepage is very slow. I am willing to pay for it as well if the OCR is good.
Openrouter.ai
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#240> These models are enabled on day one for Qualcomm and MediaTek hardware and optimized for Arm processors. Do they require GPU or can they be deployed on VPS with dedicated CPU?
Doesn't require a GPU, it will just be faster with a GPU.