What mobile devices can the smaller models run on? iPhone, Android?
apparently so, but im trying to find a working example / some details on what specific iOS / android devices are capable of running this
Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
121–130 of 343 posts
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#122Earlier quoted context omitted.
Do they tell you what training data they use for alignment? As in, what biases they intentionally put in the system they’re widely deploying?
Do you have some concrete example of biases in their models? Or are you just fishing for something to complain about?
Google’s and OpenAI often answered far-left, Progressive, and atheist. Google’s was censoring white people at one point. Facebook seems to espouse similar values. They’ve funded work to increase those values. Many mention topics relevant to these things in their papers in the bias or alignment sections.
These political systems don’t represent the majority of the world. They might not even represent half the U.S.. People relying on these A.I.’s might want to know if the A.I.’s are being intentionally trained to promote their creators’ views and/or suppress dissenters’ views. Also, people from multiple sides of the political spectrum should review such data to make sure it’s balanced.
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#123I'm currently fighting with a fastapi python app deployed to render. It's interesting because I'm struggling to see how I encode the image and send it using curl. Their example sends directly from the browser and uses a data uri.
But, this is relevant because I'm curious how this new model allows image inputs. Do you paste a base64 image into the prompt?
It feels like these models can start not only providing the text generation backend, but start to replace the infrastructure for the API as well.
Can you input images without something in front of it like openwebui?
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#124Earlier quoted context omitted.
llm is Simon's command line front-end to a lot of the llm apis, local and cloud-based. Along with aider-chat, it's my main interface to any LLM work -- it works well with a chat model, one-off queries, and piping text or output into a llm chain. For people who live on the command line, or are just put-off by web interfaces, it's a godsend. About the only thing I need to look further abroad for is when I'm working mul…
I've only used ollama over cli. As per the parent poster -- do you know if there are advantages over ollama for CLI use? Have you used both?
The llm CLI is much more unixy, letting you pipe data in and out easily. It can use hosted and local models, including ollama.
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#125Earlier quoted context omitted.
Do you have some concrete example of biases in their models? Or are you just fishing for something to complain about?
Google’s and OpenAI often answered far-left, Progressive, and atheist. Google’s was censoring white people at one point. Facebook seems to espouse similar values. They’ve funded work to increase those values. Many mention topics relevant to these things in their papers in the bias or alignment sections. These political systems don’t represent the majority of the world. They might not even represent half the U.S.. Peo…
this provocative parent-post may or may not be accurate, but what is missing IMHO is any characterization of the question asked or other context of use.. lacking that basic part to the inquiry, this statement alone is clearly amateurish, zealous and as said, provocative. Fighting in words is too easy! like falling off a log, as they say.. in politics it is almost unavoidable. Please, not start fires.
All that said yes, there are legitimate questions and there is social context. This forum is worth better questions.
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#126Earlier quoted context omitted.
Google’s and OpenAI often answered far-left, Progressive, and atheist. Google’s was censoring white people at one point. Facebook seems to espouse similar values. They’ve funded work to increase those values. Many mention topics relevant to these things in their papers in the bias or alignment sections. These political systems don’t represent the majority of the world. They might not even represent half the U.S.. Peo…
this provocative parent-post may or may not be accurate, but what is missing IMHO is any characterization of the question asked or other context of use.. lacking that basic part to the inquiry, this statement alone is clearly amateurish, zealous and as said, provocative. Fighting in words is too easy! like falling off a log, as they say.. in politics it is almost unavoidable. Please, not start fires. All that said ye…
I don’t have time to reproduce them. Fortunately, it’s easy for them to show how open and fair they are by publishing all training data. They could also publish the unaligned version or allow 3rd-party alignment.
Instead, they’re keeping it secret. That’s to conceal wrongdoing. Copyright infringement more than politics but still.
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#127If anyone else is looking for the bigger models on ollama and wondering where they are, the Ollama blog post answered that for me. The are "coming soon" so they just aren't ready quite yet[1]. I was a little worried when I couldn't find them but sounds like we just need to be patient. [1]: https://ollama.com/blog/llama3.2
ollama is "just" llama.cpp underneath, I recommend switching to LM Studio or Jan, they don't have this issue of proprietary wrapper that obfuscates, you can just use any ol GGUF
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#128Earlier quoted context omitted.
llm is Simon's command line front-end to a lot of the llm apis, local and cloud-based. Along with aider-chat, it's my main interface to any LLM work -- it works well with a chat model, one-off queries, and piping text or output into a llm chain. For people who live on the command line, or are just put-off by web interfaces, it's a godsend. About the only thing I need to look further abroad for is when I'm working mul…
I've only used ollama over cli. As per the parent poster -- do you know if there are advantages over ollama for CLI use? Have you used both?
Ollama can’t talk to OpenAI / Anthropic / etc. LLM gives you a single interface that can talk to both hosted and local models.
It also logs everything you do to a SQLite database, which is great for further analysis.
I use LLM and Ollama together quite a bit, because Ollama are really good at getting new models working and their server keeps those models in memory between requests.
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#129I'm absolutely amazed at how capable the new 1B model is, considering it's just a 1.3GB download (for the Ollama GGUF version). I tried running a full codebase through it (since it can handle 128,000 tokens) and asking it to summarize the code - it did a surprisingly decent job, incomplete but still unbelievable for a model that tiny: https://gist.github.com/simonw/64c5f5b111fe473999144932bef42... More of my notes he…
What are people using to check token length of code bases? I'd like to point certain app folders to a local LLM, but no idea how that stuff is calculated? Seems like some strategic prompting (eg: this is a rails app, here is the folder structure with file names, and btw here are the actual files to parse) would be more efficient than just giving it the full app folder? No point giving it stuff from /lib and /vendor f…
I use my https://github.com/simonw/ttok command for that - you can pipe stuff into it for a token count.
Unfortunately it only uses the OpenAI tokenizers at the moment (via tiktoken), so counts for other models may be inaccurate. I find they tend to be close enough though.
Re: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
#130Earlier quoted context omitted.
Llama 3.2 vision models don't seem that great if they have to compare them to Claude 3 Haiku or GPT4o-mini. For an open alternative I would use Qwen-2-72B model, it's smaller than the 90B and seems to perform quite better. Also Qwen2-VL-7B as an alternative to Llama-3.2-11B, smaller, better in visual benchmarks and also Apache 2.0. Molmo models: https://huggingface.co/collections/allenai/molmo-66f379e6fe3... , also s…
What interface do you use for a locally-run Qwen2-VL-7B? Inspired by Simon Willison's research[1], I have tried it out on Hugging Face[2]. Its handwriting recognition seems fantastic, but I haven't figured out how to run it locally yet. [1] https://simonwillison.net/2024/Sep/4/qwen2-vl/ [2] https://huggingface.co/spaces/GanymedeNil/Qwen2-VL-7B
MiniCPM-V 2.6 is based on Qwen 2 and is also great at handwriting. It works locally with KoboldCPP. Here are the results I got with a test I just did.
Image:
Output:
* https://pastebin.com/RKvYQasi
OCR script used:
* https://github.com/jabberjabberjabber/LLMOCR/blob/main/llmoc...
Model weights: MiniCPM-V-2_6-Q6_K_L.gguf, mmproj-MiniCPM-V-2_6-f16.gguf
Inference:
* https://github.com/LostRuins/koboldcpp/releases/tag/v1.75.2