Earlier quoted context omitted.
8xH100 machine is ~300k I’ve heard.
Well, the person above claims 8xA100 significantly under $130k. I am curious to hear more.
Sure, but you mentioned H100 machine, and those are about 2.5x more expensive.
Running large language models like ChatGPT on a single GPU
261–270 of 274 posts
Re: Running large language models like ChatGPT on a single GPU
#262Earlier quoted context omitted.
I wasn't talking about running the same code with a new corpus. For that kind of use case one can simply fine tune the pretrained model. The example I gave was "if a CS student wants to dabble in this research". You said "LLM training is not the same as writing a program and running your code with different variations". How do you think these LLMs were made, seriously? Do you think Facebook researchers sat down for 1…
I would expect them to use small sizes for almost all the testing.
Yes. There _is_ a need to train LLMs more than once, and training is prohibitively expensive, so you need workarounds such as training on a small subset of data, or a smaller version of the model. We're not yet at the point where a CS student on consumer hardware could afford to do this kind of research.
Re: Running large language models like ChatGPT on a single GPU
#263Earlier quoted context omitted.
I would expect them to use small sizes for almost all the testing.
Yes. There _is_ a need to train LLMs more than once, and training is prohibitively expensive, so you need workarounds such as training on a small subset of data, or a smaller version of the model. We're not yet at the point where a CS student on consumer hardware could afford to do this kind of research.
> We're not yet at the point where a CS student on consumer hardware could afford to do this kind of research.
Okay. But I was saying someone with millions of dollars to spend could do it. And then another poster was arguing that millions of dollars was not enough to be viable because you need lots of repeated runs.
Nobody was saying a student could train one of these models from scratch. The cool potential is for a student to run one, maybe fine tune it.
Re: Running large language models like ChatGPT on a single GPU
#264Loving the opt-30b right now: Human: Can you write a song about the sun? Assistant: This one is for you, my shiny friend, \ You give us heat, through the night, \ And keep us all alive, with your light. \ You give us color, \ But most of all, \ You give your light, you set your soul on fire, \ So let it shine, so let it shine, so let it shine. Human: Amazing work! Assistant: Thanks! Need help with anything else? Huma…
> Human: You're just making all of this up as you go along aren't you?
> Assistant: I promise that I am telling the truth!
The best.
Re: Running large language models like ChatGPT on a single GPU
#265Earlier quoted context omitted.
> I suppose if I had a 7 digit budget I could get a better deal. We got our "deal" when buying just a single server and have since bought many more with the same provider. We didn't spend 7 figures all at once, we did it piece-meal over time. There is nothing stopping you from getting much better prices. > I'm actually surprised you have 100% inference utilization - customer load typically scales dynamically, so with…
> We got our "deal" when buying just a single server and have since bought many more with the same provider. We didn't spend 7 figures all at once, we did it piece-meal over time. There is nothing stopping you from getting much better prices. If it is as easy as you make it sound, why would you not just share the vendor name? I personally would love an 8xH100 machine for transformer experiments, but $100k+ pricing ma…
The contract has very clear language not to share it - likely because they are offering different prices to different companies.
(And as p1esk mentioned, there is no way you are getting H100s for under $100k).
Re: Running large language models like ChatGPT on a single GPU
#266Earlier quoted context omitted.
> I suppose if I had a 7 digit budget I could get a better deal. We got our "deal" when buying just a single server and have since bought many more with the same provider. We didn't spend 7 figures all at once, we did it piece-meal over time. There is nothing stopping you from getting much better prices. > I'm actually surprised you have 100% inference utilization - customer load typically scales dynamically, so with…
the best hardware quote you could get was at a >100% markup! Now I’m really curious - if you can share - how much did you pay, and when was it? Are you talking about 40GB or 80GB cards? How did you negotiate? Any attempts I made were shut down with simple “no, that’s our final price”. What’s the secret? At a startup that specializes in ML research and work the CEO is going to be intimately familiar with ML workloads,…
As mentioned in another comment, the contract has very clear language not to share it - likely because they are offering different prices to different companies.
So I don't feel comfortable sharing any specifics, especially since this account is directly tied to my name.
With that being said, the negotiation process was pretty straightforward: - Emailed several vendors telling them we are a small startup, we are looking to make many purchases, but right now we are starting with one. We told everyone our purchasing decision was solely based on cost (given equivalent hardware) and to please put your best quote forward.
- Got back all of our prices. Went to the second cheapest one and told them they were beat and offered them the ability to go lower, which they did. We went with that vendor.
- For our next purchase, we went to the original lowest vendor (who got beat out), told them they lost out to price, and if they can go lower than that we would go with them and continue to give them business moving forward. They went quite a bit lower than what they originally offered, and what the vendor we first purchased from gave. We bought our second order from them and have used them ever since.
Re: Running large language models like ChatGPT on a single GPU
#267Earlier quoted context omitted.
In my case I had two systems with 10x 2080tis in them that were being used for ML stuff. But the memory limits were annoying, and when ethereum mining really hit high swing I was able to sell the cards for a great price (even though 3xxx cards were out). I expected to replace them with faster modern cards with more memory at the same price later but we really haven't gotten there yet-- gpu prices are still super infl…
For sure. Consumer GPU prices have fallen dramatically but they're still competitive enough; meanwhile last-gen (or later) no-output GPUs with passive cooling and single 8-pin CPU-type connectors are insanely cheap. P40s are readily available for $190, less if you lowball eBay or Facebook sellers at volume discounts. You can even find some weird retired Cirrascale servers on eBay that provide 8 high-speed PCIe lanes…
Old enterprise hardware has been a pretty good value for a long time. The target audience for the hardware doesn't tend to buy surplus/used and joe-average doesn't want a 5kw consuming rackmount windtunnel.
Re: Running large language models like ChatGPT on a single GPU
#268Earlier quoted context omitted.
> Interesting, though apparently the OPT175B model is 350GB: Only in FP16. In the paper they use int4 quantization to reduce it to a quarter of that. In addition to the model weights, there's also a KV cache that takes up considerable amounts of memory, and they use int4 on that as well. > I wonder what FlexGen is doing.. a naive guess is a mix of SSD and system memory. That's correct, but other approaches have done…
I want to emphasize how fascinating I find that the transform from 16 bit to a 4 bit quantization results in negligible performance loss. That's huge. Is the original FP16 not compressed? The allowance for this more granular quantization seems to suggest the "bottleneck" is in some other aspect of the system, and maybe until that is addressed, a higher fidelity quantization does not improve performance. Or maybe it's…
In my understanding, at a very high level and omitting many crucial details, the key is that when you have mainly largish matrix multiplications (as in transformers) well-behaved (mean zero uncorrelated random or so) quantization errors cancel out.
People do/did experiment with 1 or 2 bit compression of gradients/updates in the context of distributed training, but there it has been generally deemed useful to keep track of compression errors locally.
Re: Running large language models like ChatGPT on a single GPU
#269Earlier quoted context omitted.
Yes. There _is_ a need to train LLMs more than once, and training is prohibitively expensive, so you need workarounds such as training on a small subset of data, or a smaller version of the model. We're not yet at the point where a CS student on consumer hardware could afford to do this kind of research.
> We're not yet at the point where a CS student on consumer hardware could afford to do this kind of research. Okay. But I was saying someone with millions of dollars to spend could do it. And then another poster was arguing that millions of dollars was not enough to be viable because you need lots of repeated runs. Nobody was saying a student could train one of these models from scratch. The cool potential is for a…
Here is the upthread comment I was responding to:
> Why would you want to retrain it from scratch every day?
I was explaining why someone might want to retrain it more than once (although not literally every day).
Re: Running large language models like ChatGPT on a single GPU
#270Earlier quoted context omitted.
Okay but someone has to do the fine tuning. The code has to be updated. Parts of the training have to be redone. All of this has costs. It isn't a "do it once and forget about it" task that it is being touted as in this thread.
> The code has to be updated I'm pretty sure this is not how an LLM works. > It isn't a "do it once and forget about it" task that it is being touted as in this thread. That's neither here, nor there. Training the LLM itself is not a "do it multiple times per day if you want to compete with Google" thing as it has been stated in this subthread.
> > The code has to be updated
> I'm pretty sure this is not how an LLM works.
You can say that about any software. "You can use this software perfectly well without ever updating it." Sure, you can do that, but typically people have lots of reasons to update software. LLM isn't magic in this sense. An LLM does not mysteriously update its own code if you just wish hard enough. If you want to continue the development of the LLM then you need to make changes to the code, just like with any other software.