Thanks, Ollama.
(Sorry, could not resist.)
Ollama now supports AMD graphics cards
171–180 of 234 posts
Re: Ollama now supports AMD graphics cards
#172Earlier quoted context omitted.
[flagged]
Honestly, they probably should keep a little Bjarne shrine on their desk and light a little votive candle every time they release to production.
Who are these heathens who are going to ignore the basic language design work done by Kernighan & Ritchie? And whoever invented ++. Stroustrup didn't invent the C, the ++ or the notion of ++ing things in general. His contribution here is clearly quite small.
I thought the original argument for focusing on ollama alone was quite good. Once you get into dependencies it is turtles all the way down.
Re: Ollama now supports AMD graphics cards
#173Earlier quoted context omitted.
No need to be angry about this. Tech folks should be discussing this collectively and collaboratively. There's space for everything from local models running on smartphones all the way up to OpenAI style industrialized models. Back when social networks were first coming out, I used to read lots of comments about deploying and running distributed systems. I remember reading early incident reports about hotspotting and…
Not angry. Having a discussion :-). It just amazes me how the HN crowd is more than happy with just trying out a model on their machine and calling it a day and not seeing the real picture ahead. Let ignore perf concerns for a moment. Let's say I want to run it on a shared server in the enterprise network so that any application can make use of it. Each application might want to use a model of their choosing. Ollama…
> Not sure if folks here are realizing this :-)
I’m not sure you’re capable of understanding that your needs and requirements are just that, yours.
Re: Ollama now supports AMD graphics cards
#174Earlier quoted context omitted.
This shows the value ollama provides I only need to know the model name and then run a single command
It should be fairly obvious that one can find alternative models and use them in the above command too. Look, I’m not arguing that a prebuilt binary that handles model downloading has no value over a source build and manually pulling down gguf files. I just want to dispel some of the mystery. Local LLM execution doesn’t require some mysterious voodoo that can only be done by installing and running a server runtime. I…
There are 5 commands in that README two comments up, 4 can reasonably fail (I'll give cd high marks for reliability). `make` especially is a minefield and usually involves a half-hour of searching the internet and figuring out which dependencies are a problem today. And that is all assuming someone is comfortable with compiled languages. I'd hazard most devs these days are from JS land and don't know how to debug make.
Finding the correct model weights is also a challenge in my experience, there are a lot of alternatives and it is often difficult to figure out what the differences are and whether they matter.
The README is clear that I'm probably about to lose an hour debugging if I follow it. It might be one of those rare cases where it works first time but that is the exception not the rule.
Re: Ollama now supports AMD graphics cards
#175There's a thing somewhat conspicuous in its absence - why isn't llama.cpp more directly credited and thanked for providing the base technology powering this tool? All the other cool "run local" software seems to have the appropriate level of credit. You can find llama.cpp references in the code, being set up in a kind of "as is" fashion such that it might be OK as far as MIT licensing goes, but it seems kind of petty…
ollama has made a lot of nice contributions of their own. It's a good look to give a hat tip to the great work llama.cpp is also doing, but they're strictly speaking not required to do that in their advertising any more than llama.cpp is required to give credit to Google Brain, and I think that's because llama.cpp has pulled off tricks in the execution that Brain never could have accomplished, just as ollama has had…
Newton only gave credits to "Gods". He said he stands on shoulders of Gods or something like that. But he never mentioned which Gods in particular, did he?
Re: Ollama now supports AMD graphics cards
#176Earlier quoted context omitted.
Another giveaway that it's ROCm is that it doesn't support the 5700 series... I'm really salty because I "upgraded" to a 5700XT from a Nvidia GTX 1070 and can't do AI on the GPU anymore, purely because the software is unsupported. But, as a dev, I suppose I should feel some empathy that there's probably some really difficult problem causing 5700XT to be unsupported by ROCm.
I wrote a bunch of openmp code on a 5700XT a couple of years ago, if you're building from source it'll probably run fine
[deleted]
Re: Ollama now supports AMD graphics cards
#177Re: Ollama now supports AMD graphics cards
#178Earlier quoted context omitted.
The README is pretty clear, albeit it talks about a lot of optional steps you don’t need, but it’s essentially gonna be something like: git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp make wget https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF/resolve/main/mixtral-8x7b-v0.1.Q4_K_M.gguf?download=true ./main -m ./mixtral-8x7b-v0.1.Q4_K_M.gguf -n 128
This shows the value ollama provides I only need to know the model name and then run a single command
The first 3 steps GP provided are literally just the steps for installation. The "value" you mentioned is just a packaged installer (or, in the case of Linux, apparently a `curl | sh` -- and I'd much prefer the git clone version).
On multiple occasions I've been modifying llama.cpp code directly and recompiling for my own purposes. If you're using ollama on the command line, I'd say having the option to easily do that is much more useful than saving a couple commands upon installation.
Re: Ollama now supports AMD graphics cards
#179Earlier quoted context omitted.
FWIW Ollama has no concurrency support even though llama.cpp's server component (the thing that Ollama actually uses) supports it. Besides, you can't have more than 1 model running. Unloading and loading models is not free. Again, there's a lot more and really much of the real optimization work is not in Ollama; it's in llama.cpp which is completely ignored in this equation.
Thanks! Great to know. I did not know llama.cpp could do this. It should be pretty straight forward to support, not sure why they would not do it.
I'm pretty sure their primary focus right now is to gain as much mindshare as possible and they seem to be doing a great job of it. If you look at the following GitHub metrics:
https://devboard.gitsense.com/ggerganov?r=ggerganov%2Fllama....
https://devboard.gitsense.com/ollama?r=ollama%2Follama&nb=tr...
The number of people engaging with ollama is twice that of llama.cpp. And there hasn't been a dip in people engaging with Ollama in the past 6 months. However, what I do find interesting with regards to these two projects is the number of merged pull requests. If you click on the "Groups" tab and look at "Hooray", you can see llama.cpp had 72 contributors with one or more merged pull requests vs 25 for Ollama.
For Ollama, people are certainly more interested in commenting and raising issues. Compare this to llama.cpp, where the number of people contributing code changes is double that of Ollama.
I know llama.cpp is VC funded and if they don't focus on make using llama.cpp as easy to use as Ollama, they may find themselves doing all the hard stuff with Ollama reaping all the benefits.
Full Disclosure: The tool that I used is mine.
Re: Ollama now supports AMD graphics cards
#180Earlier quoted context omitted.
Apple killed off OpenCL for their platforms when they created Metal which was disappointing. Sounds like ROCm will keep it alive but the fragmentation sucks. Gotta support CUDA, OpenCL, and Metal now to be cross-platform.
What is OpenCL? AMD GPUs support CUDA. It's called HIP. You just need a bunch of #define statements like this: #ifndef __HIP__ #include #include #else #include #include #define cudaSuccess hipSuccess #define cudaStream_t hipStream_t #define cudaGetLastError hipGetLastError #endif Then your CUDA code works on AMD.
OpenCL is a Khronos open spec for GPU compute, and what you’d use on Apple platforms before Metal compute shaders and CoreML were released. If you wanted to run early ML models on Apple hardware, it was an option. There was an OpenCL backend for torch, for example.