Dang, I don't have a laptop, can I also run it on my desktop...? ;-)
As long as it you take it and keep it on your laps while running this, should be fine.
An LLM playground you can run on your laptop
31–40 of 58 posts
Re: An LLM playground you can run on your laptop
#32Re: An LLM playground you can run on your laptop
#33Dang, I don't have a laptop, can I also run it on my desktop...? ;-)
[deleted]
Re: An LLM playground you can run on your laptop
#34Dang, I don't have a laptop, can I also run it on my desktop...? ;-)
Every desktop is a laptop if you're strong enough.
Re: An LLM playground you can run on your laptop
#35Re: An LLM playground you can run on your laptop
#36This is very neat, thanks for sharing. I was wondering about a related thing — is there a way to query a llama.cpp (or other such local model) via an API from Python? In other words, I see a lot of cool applications being built with langchain + ClosedAPI, so I’m wondering if an API call to a local model could be a drop-in replacement for the ClosedAPI call?
Re: An LLM playground you can run on your laptop
#37Re: An LLM playground you can run on your laptop
#38This is very neat, thanks for sharing. I was wondering about a related thing — is there a way to query a llama.cpp (or other such local model) via an API from Python? In other words, I see a lot of cool applications being built with langchain + ClosedAPI, so I’m wondering if an API call to a local model could be a drop-in replacement for the ClosedAPI call?
In principle you can use subprocess.run with llama.cpp (especially now that the mmap patch has landed, so model load time is negligible on subsequent runs) and then use stdin and stdout to interact with it. Multiple local sessions should work too. I’ve not tried it out yet... though I was looking for an afternoon project to work on!
Re: An LLM playground you can run on your laptop
#39This is very neat, thanks for sharing. I was wondering about a related thing — is there a way to query a llama.cpp (or other such local model) via an API from Python? In other words, I see a lot of cool applications being built with langchain + ClosedAPI, so I’m wondering if an API call to a local model could be a drop-in replacement for the ClosedAPI call?
Maybe TextSynth server? It has a REST JSON API to pretty much every major LLM running locally with minimal dependencies (no python/pytorch/CUDA). (And I see that since last week it now has an HTML GUI too.) https://bellard.org/ts_server/
However, the GPU version is only available commercially. I'd like to see someone compare the speed of the CPU version against PyTorch or llama.cpp. (Edit: llama.cpp's author wrote "I expect LibNC [used by ts_server] will be better in every aspect: performance, accuracy, determinism. But hopefully with time we will close the gap." [1])
EDIT: But if you wish, here's a Python interface to llama.cpp: https://github.com/PotatoSpudowski/fastLLaMa