201–210 of 234 posts
I wish AMD did well in the Stable Diffusion front because AMD is never greedy on VRAM. The 4060Ti 16GB(minimum required for Stable Diffusion in 2024) starts at $450. AMD with ROCm is decent on Linux but pretty bad on Windows.
I'm not sure why Ollama garners so much attention. It has limited value - used for only experimenting with models + cannot support more than 1 model at a time. It's not meant for production deployments. Granted that it makes the experimentation process super easy but for something that relies on llama.cpp completely and whose main value proposition is easy model management I'm not sure it deserves the brouhaha people…
> what do you do after the initial experimentation? you need to deploy these models eventually to production
I built GaitAnalyzer[1], to analyze my gait laptop; I had deployed it briefly in production when I had enough credits to foot the AWS GPU bills. Ollama made it very simple to deploy the application, Anyone who has used docker before can now run GaitAnalyzer in their computer.
It's pretty funny to see this blog post, when I have been running Ollama on my AMD RX 6650 for weeks :D They have shipped ROCm containers since 0.1.27 (21 days ago). This blog post seems to be published along with the latest release, 0.1.29. I wonder what they actually changed in this release with regards to AMD support. Also: see this issue[0] that I made where I worked through running Ollama on an AMD card that the…
Earlier quoted context omitted.
Sad to see that the cut off is just after 6700 XT which is what is in my desktop. They indicate more devices are coming, hopefully that includes some of the more modern all in one chips with RDNA 2/3 from AMD as well.
I’ve already been using ollama with my 6700xt just fine, you just have to set some env variable to make rocm work “unoficially” The linked page says they will support more soon, so i’m guessing this will just be integrated
Earlier quoted context omitted.
> It's not meant for production deployments. I am probably not the demographics you expect. I don’t do “production” in that sense, but I have ollama running quite often when I am working, as I use it for RAG and as a fancy knowledge extraction engine. It is incredibly useful: - I can test a lot of models by just pulling them (very useful as progress is very fast), - using their command line is trivial, - the fact tha…
> I use it for RAG and as a fancy knowledge extraction engine Curious, can you share more details about your usecase?
Earlier quoted context omitted.
This shows the value ollama provides I only need to know the model name and then run a single command
The first 3 steps GP provided are literally just the steps for installation. The "value" you mentioned is just a packaged installer (or, in the case of Linux, apparently a `curl | sh` -- and I'd much prefer the git clone version). On multiple occasions I've been modifying llama.cpp code directly and recompiling for my own purposes. If you're using ollama on the command line, I'd say having the option to easily do tha…
I stopped using C++ when Go came out, no interest in ever having to write it again.
Earlier quoted context omitted.
The answer to your question is: ollama run mixtral That's it. You're running a local LLM. I have no clue how to run llama.cpp I got Stable Diffusion running and I wish there was something like ollama for it. It was painful.
The README is pretty clear, albeit it talks about a lot of optional steps you don’t need, but it’s essentially gonna be something like: git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp make wget https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF/resolve/main/mixtral-8x7b-v0.1.Q4_K_M.gguf?download=true ./main -m ./mixtral-8x7b-v0.1.Q4_K_M.gguf -n 128
Earlier quoted context omitted.
This shows the value ollama provides I only need to know the model name and then run a single command
And what will you do after trying it? Sure, you saved a few mins in trying out a model or models. What next?
Earlier quoted context omitted.
I tried it recently and couldn't figure out why it existed. It's just a very feature limited app that doesn't require you to know anything or be able to read a model card to "do AI". And that more or less answered it.
It’s because most devs nowadays are new devs and probably aren’t very familiar with native compilation. So compiling the correct version of llama.cpp for their hardware is confusing. Compound that with everyone’s relative inexperience with configuring any given model and you have prime grounds for a simple tool to exist. That’s what ollama and their Modelfiles accomplish.