Unlike Stable Diffusion, I don't stumble upon people who actually use it. Are there examples of the output this can generate? What happens once you manage to run the model?
[deleted]
Fork of Facebook’s LLaMa model to run on CPU
131–140 of 178 posts
Re: Fork of Facebook’s LLaMa model to run on CPU
#132Earlier quoted context omitted.
Speaking for myself, I have already gotten more use out of 2 weeks of chatgpt than I have out of 16 years of Bitcoin
14. First Bitcoin was mined 14 years ago. And Bitcoins have not been mined with GPUs since 2013.
You pedantry reinforces the point, rather than diminishing it.
Re: Fork of Facebook’s LLaMa model to run on CPU
#133The thing I like the most about the current AI wave is the pressure is putting on computing hardware. Yes, mobile phones with long battery lives are cool and all of that, but most cool things I like are locked behind huge computational requirements.
Crazy to me that as soon as one GPU wave is dying (crypto), another one is picking up slack.
Charlie Stross (cstross on here) had a fun blog post[1] about this phenomenon just a week and a half ago.
> As for what you should look to invest in?
> I'm sure it's just a coincidence that training neural networks and mining cryptocurrencies are both applications that benefit from very large arrays of GPUs. [...]
> If I was a VC I'd be hiring complexity theory nerds to figure out what areas of research are promising once you have Yottaflops of numerical processing power available, then I'd be placing bets on the GPU manufacturers going there
[1]: https://www.antipope.org/charlie/blog-static/2023/02/place-y...
Re: Fork of Facebook’s LLaMa model to run on CPU
#134Earlier quoted context omitted.
Agree. I work in robotics and we never have enough compute. I want to see us get to the point where the most advanced robot ever has all the compute it needs onboard, and that means huge growth in compute density and efficiency are needed.
That's genuinely surprising. What sort of on-board compute do you typically have today?
a common example from my robotics experience (mainly mobile robots) has been getting something powerful enough to run our image recognition/interpreting sensor data. We often have something like several microprocessors (think:arduino equivalent running c++ or c) which run all the motor control etc and a high level system (used to often be raspberry pi, now more often nvidia jetson nano) listening to all of those and using most of it's computing power on some kind of sensor data, usually image recognition or processing TOF camera/lidar/radar data etc. We often have to optimise hard to get a couple of cycles or "frames" per second with these, which really puts limitations on how robots respond (250ms delay is veeeery noticable, especially if it's in obstacle avoidance - relatively common)
Re: Fork of Facebook’s LLaMa model to run on CPU
#135Unlike Stable Diffusion, I don't stumble upon people who actually use it. Are there examples of the output this can generate? What happens once you manage to run the model?
I have been using similar models like LLM for helping draft fictional stories. The community fine tuned models are geared towards SFW and/or NSFW story competition.
https://github.com/KoboldAI/KoboldAI-Client To read more about current popular models.
https://koboldai.net/ is a way to run some of these models in the "cloud". There's no account required and the prompts are run on other people's hardware, with priority weighting based on how much compute you have used or donated. There's an anonymous api key and there's no expectation that the output can't be logged.
The models that run on hardware locally are very basic in the quality of output. Here's an example of a 6B output used to try to emulate chatgpt. https://mobile.twitter.com/Knaikk/status/1629711223863345154 The model was finetuned on story completion so it's not meaningfully comparable.
It's less popular because the hardware required for the great output is still above the top of line consumer specs. 24 gb vram is closer to a bare minimum to get meaningful output, and fine-tuning is still out of reach. There's some development with using services like runpod.
Re: Fork of Facebook’s LLaMa model to run on CPU
#136Earlier quoted context omitted.
The download size is large but conda doesn't ruin any existing configuration unless you explicitly tell it to be your native python environment. Conda is set up as a self-contained set of independent environments. Why would your system care what's inside the Anaconda directory unless you explicitly add it to your PATH/bash?
I haven't touched that steaming pile of shite in a looong while, so - who knows - they might have managed to minimize the amount of havoc their wreak on their user's systems. But ... I seem to recall ... Conda tries to install GPU drivers does is not? ... Is that not the case anymore? Because if it still does, your theory about "Why would your system care" and all that doesn't really hold water.
I use miniconda on Linux but it's never attempted to install graphics drivers on Windows.
Re: Fork of Facebook’s LLaMa model to run on CPU
#137Earlier quoted context omitted.
You don't need 256 GB. A pair of the new 48GB DDR5 will work along with a pair of 32GB sticks should work in a consumer DDR5 MB to fit the weights. It does burst when initially loading. So, a fast disk with about the same swap size as RAM seems necessary. It took about 25 mins to generate a single 500 character response using a 5800X & 32 GB DDR4, but I was not able to get to it to run on more than 1 thread with the…
Why? Is it a limitation of the model or just something with the configuration that you couldn't figure out for this test?
I tried mark's OMP_NUM_THREADS suggestion (https://news.ycombinator.com/item?id=35018559), did not see any an obvious change to make it parallel, and given the huggingface patch (https://github.com/huggingface/transformers/pull/21955) once it gets in is suppose to allow streaming from RAM to the GPU. So, for me it was not worth the effort to keep working on the CPU version as even the best case ~30X speedup will still take around a minute to run the 7B.
Re: Fork of Facebook’s LLaMa model to run on CPU
#138Unlike Stable Diffusion, I don't stumble upon people who actually use it. Are there examples of the output this can generate? What happens once you manage to run the model?
ive used LLMS a lot for filling out details in my dnd worlds. Both openai products but also the open source GPT-J from euluther Things like writing the text of some books for players to read, of I have to curate just like people do with stable diffusion. Also used it to write songs, its surprisingly good at taking things like chord progressions written in notation and rolling with variations on them
Re: Fork of Facebook’s LLaMa model to run on CPU
#139It's useless before the model gets instruction and preference tunings. Won't even follow a simple ask, it will just assume it is a list of questions and generate more, or continue with slightly related comments. FB trained a LLaMA-I (instruction tuned) variant for sports, just to show they can, but I don't think it got released.
it's not that useless, you just have to prompt it the right way (usually by offering an example of the kind of output you want)
Re: Fork of Facebook’s LLaMa model to run on CPU
#140Earlier quoted context omitted.
Someone must have trained an LLM for that for sure.
Oh yes “”” Hackernews senator: “”Someone on the internet said meta aka Facebook is not considered a real data native, clean coder and high IQ company unless your new language model exceeds the elegance and slipperiness of mark Zuckerbergs (you) language output in senate hearings. he is smoother than a lake in the metaverse.“” Mark LLM: “ Yes, unfortunately, the media and our competitors are all over the idea that Met…
I have to say "he is smoother than a lake in the metaverse" is presumably accidental, based on the quality of the rest of that text, but it has to be one of the wittiest phrases ive seen LLMs come out with to date