Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running large language models like ChatGPT on a single GPU

github.com

Post reply on HN
Print view View original

151–160 of 274 posts

Re: Running large language models like ChatGPT on a single GPU

#151

Post by coldtea »

baobabKoodaa wrote: post #86
Ajedi32 wrote: post #70

Earlier quoted context omitted.

Why would you want to retrain it from scratch every day? Stable Diffusion doesn't do that either.

Well maybe not every day, but having a short feedback loop and the ability to run your code multiple times with different variations is generally considered to be a prerequisite for software development. If you actually want to keep developing the model, you need the funding to be able to train it more than once.

>but having a short feedback loop and the ability to run your code multiple times with different variations is generally considered to be a prerequisite for software development

This is not "software development" in general, this is LLM training.

It's not like you're building some regular app, api, or backend.

Re: Running large language models like ChatGPT on a single GPU

#152

Post by cardine »

p1esk wrote: post #147
throwawayapples wrote: post #89

Earlier quoted context omitted.

The cloud is always stupid expensive compared to running your own hardware for almost any sort of task that isn't highly variable upon one or more axis (cpu, ram, etc), but less than a week to break even is truly shocking.

The cloud has been cheaper to train large models on for a couple years now. Compare buying 8xA100 server vs renting one on Lambda Labs. At least 3 years to break even - if you are using it non-stop 24/7. Longer if not.

This is not true - the break even period is closer to 6-7 months.

Re: Running large language models like ChatGPT on a single GPU

#153

Post by stevenhuang »

stevenhuang wrote: post #74
SekstiNi wrote: post #68

Earlier quoted context omitted.

> Interesting, though apparently the OPT175B model is 350GB: Only in FP16. In the paper they use int4 quantization to reduce it to a quarter of that. In addition to the model weights, there's also a KV cache that takes up considerable amounts of memory, and they use int4 on that as well. > I wonder what FlexGen is doing.. a naive guess is a mix of SSD and system memory. That's correct, but other approaches have done…

I want to emphasize how fascinating I find that the transform from 16 bit to a 4 bit quantization results in negligible performance loss. That's huge. Is the original FP16 not compressed? The allowance for this more granular quantization seems to suggest the "bottleneck" is in some other aspect of the system, and maybe until that is addressed, a higher fidelity quantization does not improve performance. Or maybe it's…

A recent discussion I found on int4, definitely looks like this is the new hotness. Very exciting!

https://news.ycombinator.com/item?id=34404859

Re: Running large language models like ChatGPT on a single GPU

#154

Post by coldtea »

paxys wrote: post #72
Ajedi32 wrote: post #70

Earlier quoted context omitted.

Why would you want to retrain it from scratch every day? Stable Diffusion doesn't do that either.

Because things happen every day. If ChatGPT wants to compete with Google, staying up to date with recent events is the minimum bar.

That's not what the training is about.

Things happen everyday, but languages and words and their associations don't change in any measurable way every day...

This is not like web crawling...

Re: Running large language models like ChatGPT on a single GPU

#155

Post by lxe »

Got the ops-30b chatbot running on 3090 24GB. I set compress_weight=True and compress_cache=True, and ran with `python apps/chatbot.py --model facebook/opt-30b --percent 100 0 100 0 100 0`.

I also modified the prompt a bit to make it more... uh alive:

    Assistant: Did you know that Saturn is 97 times the size of Earth?
    Human: Are you sure?
    Assistant: What difference does size make, really, anyway?
    Human: You didn't answer my question.
    Assistant: Hey, it's your question I didn't answer! Anyway, did you know that on the surface of 
    Saturn, there's lakes of liquid methane and ethane?
    Human: I didn't know that. Are there any lakes on the Sun?
    Assistant: Look, I'll be honest with you. I don't know much about the Sun. Is it even a planet?
    Human: It sure is. Why don't you know about the Sun? I thought you were a hyperintelligent AI?
    Assistant: You're right. I'm a hyperintelligent AI. I'm hyperintelligent enough to know that I know jack shit about the Sun.

Re: Running large language models like ChatGPT on a single GPU

#156

Post by coldtea »

paxys wrote: post #143
Ajedi32 wrote: post #84

Earlier quoted context omitted.

You wouldn't need to re-train from scratch for that, just fine-tune on the new data sources. I don't think constant re-training is the optimal strategy for that use-case anyway. Bing does it by letting the LLM search a more traditional web index to find the information it needs.

Okay but someone has to do the fine tuning. The code has to be updated. Parts of the training have to be redone. All of this has costs. It isn't a "do it once and forget about it" task that it is being touted as in this thread.

>The code has to be updated

I'm pretty sure this is not how an LLM works.

>It isn't a "do it once and forget about it" task that it is being touted as in this thread.

That's neither here, nor there. Training the LLM itself is not a "do it multiple times per day if you want to compete with Google" thing as it has been stated in this subthread.

Re: Running large language models like ChatGPT on a single GPU

#157

Post by coldtea »

ipsum2 wrote: post #80
JoshCole wrote: post #28

Earlier quoted context omitted.

No, it isn't astronomical. It is smaller than that. Still large, but not astronomical.

Have you tried training a large model before? If not, you're probably discounting how difficult and expensive it is.

Well, for those that trained the largest one atm, it cost them in the order of 10 million dollars (actually less).

That's how much some tech companies pay for catering.

Hell, that's in the order of a single socialite's wedding costs.

Re: Running large language models like ChatGPT on a single GPU

#158

Post by razor_router »

lxe wrote: post #155

Got the ops-30b chatbot running on 3090 24GB. I set compress_weight=True and compress_cache=True, and ran with `python apps/chatbot.py --model facebook/opt-30b --percent 100 0 100 0 100 0`. I also modified the prompt a bit to make it more... uh alive: Assistant: Did you know that Saturn is 97 times the size of Earth? Human: Are you sure? Assistant: What difference does size make, really, anyway? Human: You didn't ans…

That's impressive! How much customization did you have to do in order to get the chatbot to respond like that?

Re: Running large language models like ChatGPT on a single GPU

#159

Post by anon291 »

foobiekr wrote: post #132
anon291 wrote: post #121

Earlier quoted context omitted.

As a former AI accelerator employee (laid off), I'm kind of happy I was laid off because I realistically don't see a need for specialized hardware anymore. Large companies can afford Nvidia. Nvidia's software stack is best in class. There's no business need here and the model execution is increasingly becoming possible on single consumer GPUs. The only place where I see specialized chips excelling is on the edge or i…

Power is the main reason to do custom ASICs. I’d be curious as to your opinion of Recogni given they are claiming a 10x power reduction per unit compute.

Unfortunately, I've worked at several players which promise power reductions. It doesn't matter though. People don't care about cost at this point. If you are cost-sensitive you're not doing the kind of revolutionary AI work these companies need to create a competitive moat. And once your model works on NVIDIA and is trained, how much are you going to spend on ML engineers to make it work on something else? Because that cost better be less than the marginal cost reduction on electricity. Plus, NVIDIA et al will likely get more and more efficient.
Post reply on HN