21–30 of 65 posts
Earlier quoted context omitted.
Are you talking about the Hugging Face Python libraries, the Hugging Face hosted inference APIs, the Hugging Face web interfaces, the Hugging Face iPhone app, Hugging Face Spaces (hosted Docker environments with GPU access) or something else?
I updated my comment above: I’m using HF transformers repo, which gets models from HF hub.
Earlier quoted context omitted.
I updated my comment above: I’m using HF transformers repo, which gets models from HF hub.
Do you have an NVIDIA GPU? I have not had much luck with the transformers library on a Mac.
Earlier quoted context omitted.
Do you have an NVIDIA GPU? I have not had much luck with the transformers library on a Mac.
Of course. I thought Nvidia GPUs are pretty much a must have to play with DL models.
I wonder if Ollama will or plans to have other "Supported backends" than llama.cpp. It's listed on the very last line of their readme as if the llama.cpp dependency is just incidental and a very minor detail rather than Ollama as a deployment mechanism for llama.cpp and gguf based models.
All of this of course acknowledging that llama.cpp is an incredible project with competitive performance and support for almost any platform.
[1] https://github.com/ml-explore/mlx
Earlier quoted context omitted.
Of course. I thought Nvidia GPUs are pretty much a must have to play with DL models.
Well being able to run these models on CPU was pretty much the revolutionary part of llama.cpp.
I wonder if Ollama will or plans to have other "Supported backends" than llama.cpp. It's listed on the very last line of their readme as if the llama.cpp dependency is just incidental and a very minor detail rather than Ollama as a deployment mechanism for llama.cpp and gguf based models.
this looks very awesome. can someone tell me why there is no chatter about this? is there something else out there that blows this out of the water in terms of ease of use and access to sample many LLM's ?
Earlier quoted context omitted.
Do you have an NVIDIA GPU? I have not had much luck with the transformers library on a Mac.
Of course. I thought Nvidia GPUs are pretty much a must have to play with DL models.
Earlier quoted context omitted.
Of course. I thought Nvidia GPUs are pretty much a must have to play with DL models.
Ollama supports many radeons now. And I guess llama.cpp does too, after all it's what ollama uses as backend.
I wonder if Ollama will or plans to have other "Supported backends" than llama.cpp. It's listed on the very last line of their readme as if the llama.cpp dependency is just incidental and a very minor detail rather than Ollama as a deployment mechanism for llama.cpp and gguf based models.
I don't think they will move away from llama.cpp until they are forced to. The number of people contributing to llama.cpp is quite significant [1] and it wouldn't make sense to use another backend given how quickly llama.cpp is iterating and growing. [1] https://devboard.gitsense.com/ggerganov?r=ggerganov%2Fllama.... Full disclosure: This is my tool
The chance onnx becomes significantly relevant here went from 1% to 15% this week. They're demo'ing ~2x faster inference with Phi-3. There's been fits and starts on LLMs in ONNX for a year, but, with Wintel's AI PC™ push, and all the constituent parts in place (4 bit quants! adaptive quants!), I'd put very good money on it.