41–50 of 58 posts
This is very neat, thanks for sharing. I was wondering about a related thing — is there a way to query a llama.cpp (or other such local model) via an API from Python? In other words, I see a lot of cool applications being built with langchain + ClosedAPI, so I’m wondering if an API call to a local model could be a drop-in replacement for the ClosedAPI call?
I'm on mobile and can't find it right now though.
This is very neat, thanks for sharing. I was wondering about a related thing — is there a way to query a llama.cpp (or other such local model) via an API from Python? In other words, I see a lot of cool applications being built with langchain + ClosedAPI, so I’m wondering if an API call to a local model could be a drop-in replacement for the ClosedAPI call?
[1] https://leanpub.com/langchain
EDIT: GitHub repo https://github.com/mark-watson/langchain-book-examples
This is very neat, thanks for sharing. I was wondering about a related thing — is there a way to query a llama.cpp (or other such local model) via an API from Python? In other words, I see a lot of cool applications being built with langchain + ClosedAPI, so I’m wondering if an API call to a local model could be a drop-in replacement for the ClosedAPI call?
Maybe TextSynth server? It has a REST JSON API to pretty much every major LLM running locally with minimal dependencies (no python/pytorch/CUDA). (And I see that since last week it now has an HTML GUI too.) https://bellard.org/ts_server/ However, the GPU version is only available commercially. I'd like to see someone compare the speed of the CPU version against PyTorch or llama.cpp. (Edit: llama.cpp's author wrote "I…
This is very neat, thanks for sharing. I was wondering about a related thing — is there a way to query a llama.cpp (or other such local model) via an API from Python? In other words, I see a lot of cool applications being built with langchain + ClosedAPI, so I’m wondering if an API call to a local model could be a drop-in replacement for the ClosedAPI call?
Maybe TextSynth server? It has a REST JSON API to pretty much every major LLM running locally with minimal dependencies (no python/pytorch/CUDA). (And I see that since last week it now has an HTML GUI too.) https://bellard.org/ts_server/ However, the GPU version is only available commercially. I'd like to see someone compare the speed of the CPU version against PyTorch or llama.cpp. (Edit: llama.cpp's author wrote "I…
Earlier quoted context omitted.
I wonder how people that don't read properly before doing something will fare in a world of text interfaces/ai.
Better? Because the AI will help them?
Dang, I don't have a laptop, can I also run it on my desktop...? ;-)
Can it run Crysis?
...It is a good AGI test - trying the consistency in the "spacial" and temporal development of the construed world.
Earlier quoted context omitted.
I don’t know about out of the box, but I’ve been having a great time training my own domain-specific models from scratch and utilizing them locally. It does seem that the more domain specific a task is, far fewer params are needed. Having fun right now trying to build a model in GDELT, not much luck so far, but I’ve pushed less than 5% of the data through so far. I’ve also been experimenting on fine-tuning Llama on m…
What is the ballpark cost of this hobby, if I may ask? Do you train exclusively in the cloud?
I do a lot of CPU and sharded training too. I really wish there were better options for hobbyists to play around with this stuff.
Earlier quoted context omitted.
I don’t know about out of the box, but I’ve been having a great time training my own domain-specific models from scratch and utilizing them locally. It does seem that the more domain specific a task is, far fewer params are needed. Having fun right now trying to build a model in GDELT, not much luck so far, but I’ve pushed less than 5% of the data through so far. I’ve also been experimenting on fine-tuning Llama on m…
Training domain-specific models from scratch is both very difficult and also not possible in some cases. Sometimes we want to generate some niche content where there isn't enough training data to create a domain-specific model. We would have to somehow find a model that can produce the niche content through understanding what we're asking rather than through actually being trained on it. Producing "understanding" is…
An LLM playground whose UI you can run on your laptop.
No, this is running LLMs on your laptop, although it can also connect to remote ones. Eg: "Automatically detects local models in your HuggingFace cache, and lets you install new ones." Llamma.cpp and HuggingFace models are all local.