> These models are enabled on day one for Qualcomm and MediaTek hardware and optimized for Arm processors.
Do they require GPU or can they be deployed on VPS with dedicated CPU?
Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
201–210 of 343 posts
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#202meta.ai still running on 3.1
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#203I'm absolutely amazed at how capable the new 1B model is, considering it's just a 1.3GB download (for the Ollama GGUF version). I tried running a full codebase through it (since it can handle 128,000 tokens) and asking it to summarize the code - it did a surprisingly decent job, incomplete but still unbelievable for a model that tiny: https://gist.github.com/simonw/64c5f5b111fe473999144932bef42... More of my notes he…
If you are in US, you get 1 billion tokens a DAY with Gemini (Google) completely free of cost.
Gemini Flash is fast with upto 4 million token context.
Gemini Flash 002 improved in math and logical abilities surpassing Claude and Gpt 4o
You can simply use Gemini Flash for Code Completion, git review tool and many more.
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#204Newbie question, what size model would be needed to have a 10x software engineer skills and no knowledge of the human kind (ie, no need to know how to make a pizza or sequence your DNA). Is there such a model?
Not yet. But, Nvidia's CEO announced a few months ago that we're about 5 years away. And, OpenAI just this week announced Super Intelligence is up to 2000 days (e.g. around 5 years) away.
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#205Earlier quoted context omitted.
Llama 3.2 vision models don't seem that great if they have to compare them to Claude 3 Haiku or GPT4o-mini. For an open alternative I would use Qwen-2-72B model, it's smaller than the 90B and seems to perform quite better. Also Qwen2-VL-7B as an alternative to Llama-3.2-11B, smaller, better in visual benchmarks and also Apache 2.0. Molmo models: https://huggingface.co/collections/allenai/molmo-66f379e6fe3... , also s…
1. Ignore the benchmarks. I've been A/Bing 11B today with Molmo 72B [1], which itself has an ELO neck-and-neck with GPT4o, and it's even. Because everyone in open source tends to train on validation benchmarks, you really can not trust them. 2. The method of tokenization/adapter is novel and uses many fewer tokens than all comparable CLIP/SigLIP-adapter models, making it _much_ faster. Attention is O(n^2) on memory/c…
I tried some OCR use cases, Claude Sonnet just blows Molmo.
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#206I'm absolutely amazed at how capable the new 1B model is, considering it's just a 1.3GB download (for the Ollama GGUF version). I tried running a full codebase through it (since it can handle 128,000 tokens) and asking it to summarize the code - it did a surprisingly decent job, incomplete but still unbelievable for a model that tiny: https://gist.github.com/simonw/64c5f5b111fe473999144932bef42... More of my notes he…
What are people using to check token length of code bases? I'd like to point certain app folders to a local LLM, but no idea how that stuff is calculated? Seems like some strategic prompting (eg: this is a rails app, here is the folder structure with file names, and btw here are the actual files to parse) would be more efficient than just giving it the full app folder? No point giving it stuff from /lib and /vendor f…
You can try Gemini Token count. https://ai.google.dev/api/tokens
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#207Can one of thse models be run on a single machine? What specs do you need?
The 8B models run fine on a M1 pro 16GB.
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#208I'm blown away with just how open the Llama team at Meta is. It is nice to see that they are not only giving access to the models, but they at the same time are open about how they built them. I don't know how the future is going to go in the terms of models, but I sure am grateful that Meta has taken this position, and are pushing more openness.
They have a hose of ad money and have nothing to lose doing this. You can’t say that for the other guys.
> They have a hose of ad money and have nothing to lose doing this.
If I didn’t have context I’d assume this was about Google.
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#209Earlier quoted context omitted.
Hoping to get this out soon w/ Ollama. Just working out a couple of last kinks. The 11b model is legit good though, particularly for tasks like OCR. It can actually read my cursive handwriting.
Naah, Qwen2-VL-7b still is much much better than 11b model for handwritten OCR from what i have tested. The 11b model hallucinates in case of handwritten OCR.
Where can I try it out. The playground on their homepage is very slow. I am willing to pay for it as well if the OCR is good.
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#210Earlier quoted context omitted.
What interface do you use for a locally-run Qwen2-VL-7B? Inspired by Simon Willison's research[1], I have tried it out on Hugging Face[2]. Its handwriting recognition seems fantastic, but I haven't figured out how to run it locally yet. [1] https://simonwillison.net/2024/Sep/4/qwen2-vl/ [2] https://huggingface.co/spaces/GanymedeNil/Qwen2-VL-7B
MiniCPM-V 2.6 is based on Qwen 2 and is also great at handwriting. It works locally with KoboldCPP. Here are the results I got with a test I just did. Image: * https://imgur.com/wg0kdQK Output: * https://pastebin.com/RKvYQasi OCR script used: * https://github.com/jabberjabberjabber/LLMOCR/blob/main/llmoc... Model weights: MiniCPM-V-2_6-Q6_K_L.gguf, mmproj-MiniCPM-V-2_6-f16.gguf Inference: * https://github.com/LostRui…
Thanks for the hint. Will try the out!