Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An LLM playground you can run on your laptop

github.com

Post reply on HN
Print view View original

51–58 of 58 posts

Re: An LLM playground you can run on your laptop

#51

Post by MacsHeadroom »

d4rkp4ttern wrote: post #36

This is very neat, thanks for sharing. I was wondering about a related thing — is there a way to query a llama.cpp (or other such local model) via an API from Python? In other words, I see a lot of cool applications being built with langchain + ClosedAPI, so I’m wondering if an API call to a local model could be a drop-in replacement for the ClosedAPI call?

Yes, there are python bindings for llama.cpp and the text-generation-webui already uses them for local inference. https://github.com/oobabooga/text-generation-webui/wiki/llam...

"pip install llamacpp" or https://github.com/thomasantony/llamacpp-python

Re: An LLM playground you can run on your laptop

#52

Post by MacsHeadroom »

holografix wrote: post #15
newswasboring wrote: post #10

Earlier quoted context omitted.

GPT2 is ancient news. There are now local running models which, allegedly, can reach the same performance as GPT-4. Look up llama.cpp[1] and all the various community generated models. [1] https://github.com/ggerganov/llama.cpp

Does llama.cop not run on GPU?

llama.cpp is CPU only but llama runs on GPU using the HuggingFace Transformers library. You can run the llama model which far outpaces GPT-3.5 on a couple of $200 Tesla P40 GPUs at faster speeds than GPT-3.5 Turbo, completely locally.

Re: An LLM playground you can run on your laptop

#54

Post by boppo1 »

d4rkp4ttern wrote: post #36

This is very neat, thanks for sharing. I was wondering about a related thing — is there a way to query a llama.cpp (or other such local model) via an API from Python? In other words, I see a lot of cool applications being built with langchain + ClosedAPI, so I’m wondering if an API call to a local model could be a drop-in replacement for the ClosedAPI call?

I think langchain lets you do this.

Re: An LLM playground you can run on your laptop

#55

Post by Lockal »

Zetobal wrote: post #6
Cyphase wrote: post #3

Earlier quoted context omitted.

It supports local models.

I wonder how people that don't read properly before doing something will fare in a world of text interfaces/ai.

They will answer "Please don't comment on whether someone read an article, please review https://news.ycombinator.com/newsguidelines.html (actually, a bad excuse, but it is what it is).

Re: An LLM playground you can run on your laptop

#57

Post by swyx »

zxexz wrote: post #48
worldsayshi wrote: post #18

Earlier quoted context omitted.

What is the ballpark cost of this hobby, if I may ask? Do you train exclusively in the cloud?

Really depends. I’ve pretty much almost burned through all the cloud credits I’ve been able to find at this point. Probably spent 20 grand over a year in credits? Never more than a couple hundred of my own money. Still trying to find a better solution. I have some 2080 TIs that I use locally. I tend to prototype architectures locally then rent out some A100s for a few hours to see how it goes, continue training when…

got a quick guide as to available cloud credits for GPUs?
Post reply on HN