Dang, I don't have a laptop, can I also run it on my desktop...?
;-)
An LLM playground you can run on your laptop
21–30 of 58 posts
Re: An LLM playground you can run on your laptop
#22Thanks for this! The compare feature is very cool, especially being able to play around with GPT4 settings (I'm still on the waiting list for GPT4 API so having access to this now is fantastic).
Re: An LLM playground you can run on your laptop
#23Earlier quoted context omitted.
Are there any good local models? Gpt-2 is pants once you've gotten used to 4.
GPT2 is ancient news. There are now local running models which, allegedly, can reach the same performance as GPT-4. Look up llama.cpp[1] and all the various community generated models. [1] https://github.com/ggerganov/llama.cpp
Incredibly difficult to measure it, but it seems llama is closest to GPT-3[0].
Which is pretty cool all things considered. Certainly better than most of us would have expected a short while ago, right?
Re: An LLM playground you can run on your laptop
#24Dang, I don't have a laptop, can I also run it on my desktop...? ;-)
By laptop they mean you don't need a beefy gaming PC with 200 RGB lights.
Re: An LLM playground you can run on your laptop
#25Earlier quoted context omitted.
Are there any good local models? Gpt-2 is pants once you've gotten used to 4.
I don’t know about out of the box, but I’ve been having a great time training my own domain-specific models from scratch and utilizing them locally. It does seem that the more domain specific a task is, far fewer params are needed. Having fun right now trying to build a model in GDELT, not much luck so far, but I’ve pushed less than 5% of the data through so far. I’ve also been experimenting on fine-tuning Llama on m…
Training domain-specific models from scratch is both very difficult and also not possible in some cases. Sometimes we want to generate some niche content where there isn't enough training data to create a domain-specific model. We would have to somehow find a model that can produce the niche content through understanding what we're asking rather than through actually being trained on it. Producing "understanding" is not something you can do at home unless you have a bunch of A100s and have studied machine learning for years.
-Emily
Re: An LLM playground you can run on your laptop
#26Earlier quoted context omitted.
GPT2 is ancient news. There are now local running models which, allegedly, can reach the same performance as GPT-4. Look up llama.cpp[1] and all the various community generated models. [1] https://github.com/ggerganov/llama.cpp
Does llama.cop not run on GPU?
It doesn't, it's CPU-only.
-Emily
Re: An LLM playground you can run on your laptop
#27Dang, I don't have a laptop, can I also run it on my desktop...? ;-)
Can it run Crysis?
Re: An LLM playground you can run on your laptop
#28As an alternative for purely local LLMs, I've been having fun with this setup: https://github.com/oobabooga/text-generation-webui
Do you know what is the way to find models compatible with oobabooga/text-generation-webui ? I downloaded one with the included script and that worked, but if I try different ones it seems there are so many formats so no idea say how do I search huggingface or google to find the correct format. I would like to try this new quantized LLAMA versions with the GUI, I can run them in the CLI on the CPU but llama.cpp uses…
https://rentry.org/nur779 (scroll down to the Huggingface ones)
Re: An LLM playground you can run on your laptop
#29Someone with more inclination than I at the moment might be able to say something interesting about this being from Nat Friedman (former CEO of GitHub).
Good to see a CEO still coding