Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running large language models like ChatGPT on a single GPU

github.com

Post reply on HN
Print view View original

251–260 of 274 posts

Re: Running large language models like ChatGPT on a single GPU

#251

Post by raihansaputra »

NonEUCitizen wrote: post #209
raihansaputra wrote: post #167

Earlier quoted context omitted.

Thing is, you don't have to totally switch to Linux. I'm running ML/CUDA workloads through WSL without too many problems.

Although not "too many," what kind of problems have you encountered running ML/CUDA in WSL? Thanks.

WSL problems but not related to CUDA:

- need a patch to expose ports for services in WSL to network (WSLHostPatcher)

- the Virtual Hard Disk (vhdx) does not free unused space easily and it can grow quickly. I ended up just symlinking my code and dataset folders to mounts, not saving a lot of data inside the vhdx

- beware of upgrades etc. I think I nuked my WSL 2x due to config issues. Having all your code/data on mounts also makes this easy.

related to ML/CUDA:

- how you install pytorch + CUDA matters. I ended up just installing from `conda --channel fastchan` and don't touch it. not ideal but it works

- Don't forget to configure the RAM allocation in case you need a lot

- I haven't tried running a CUDA Docker on WSL. May be an easier way to do this.

Running on Windows directly is also an option. I choose to run on WSL because most learning resources/documentation refer more to Linux installs and setups.

Re: Running large language models like ChatGPT on a single GPU

#252

Post by icelancer »

nullc wrote: post #213
icelancer wrote: post #200

Earlier quoted context omitted.

A lot of 2U cases won't fit a consumer GPU. Furthermore, Tesla-equivalents are usually either significantly cheaper than their consumer counterpart (for last-gen and older GPUs) or similar in price with far more RAM. I bought a bunch of Tesla P40s at a really low price compared to what 1080tis are going for.

I bought a couple dozen 24GB K80s for like $40 each.

I wanted to so bad, but for my project they just didn't work.

Doesn't mean I won't buy them anyway soon... gah, your message isn't helping me!

Re: Running large language models like ChatGPT on a single GPU

#253

Post by nullc »

icelancer wrote: post #252
nullc wrote: post #213

Earlier quoted context omitted.

I bought a couple dozen 24GB K80s for like $40 each.

I wanted to so bad, but for my project they just didn't work. Doesn't mean I won't buy them anyway soon... gah, your message isn't helping me!

In my case I had two systems with 10x 2080tis in them that were being used for ML stuff. But the memory limits were annoying, and when ethereum mining really hit high swing I was able to sell the cards for a great price (even though 3xxx cards were out). I expected to replace them with faster modern cards with more memory at the same price later but we really haven't gotten there yet-- gpu prices are still super inflated and all except the most absurd cards are still memory starved (for ML). In the interim it turned out to be really cheap to get K80s.

Re: Running large language models like ChatGPT on a single GPU

#254

Post by efreak »

exebook wrote: post #103
mryab wrote: post #56

Earlier quoted context omitted.

There is! See https://petals.ml/ for inference of models like BLOOM-176B over the internet or https://arxiv.org/abs/2301.11913 and https://arxiv.org/abs/2206.01288 that show you how to do pretraining from scratch in the same setting. Disclaimer: I'm a coauthor of these systems (including the one in OP)

Amazing work! If I had a GPU, I'd join. I know similar project for text-to-image: https://aqualxx.github.io/stable-ui/

https://github.com/aqualxx/stable-horde-notebook

My only problem with stable horde is that their anti-cp measure involves checking the prompt for words like small, meaning I can't use a nsfw-capable model with certain prompts (holding a very small bag, etc). That, and seeing great things in the image rating and being unable to reproduce because it doesn't provide the prompt.

Re: Running large language models like ChatGPT on a single GPU

#255

Post by Dylan16807 »

baobabKoodaa wrote: post #228
coldtea wrote: post #227

Earlier quoted context omitted.

> If you are claiming that training a LLM literally only one time is enough and there is no need to train it more than once, you are wrong. No, I'm rather claiming that what you claimed is wrong in the context of LLM training: "Well maybe not every day, but having a short feedback loop and the ability to run your code multiple times with different variations is generally considered to be a prerequisite for software d…

I wasn't talking about running the same code with a new corpus. For that kind of use case one can simply fine tune the pretrained model. The example I gave was "if a CS student wants to dabble in this research". You said "LLM training is not the same as writing a program and running your code with different variations". How do you think these LLMs were made, seriously? Do you think Facebook researchers sat down for 1…

I would expect them to use small sizes for almost all the testing.

Re: Running large language models like ChatGPT on a single GPU

#256

Post by icelancer »

nullc wrote: post #253
icelancer wrote: post #252

Earlier quoted context omitted.

I wanted to so bad, but for my project they just didn't work. Doesn't mean I won't buy them anyway soon... gah, your message isn't helping me!

In my case I had two systems with 10x 2080tis in them that were being used for ML stuff. But the memory limits were annoying, and when ethereum mining really hit high swing I was able to sell the cards for a great price (even though 3xxx cards were out). I expected to replace them with faster modern cards with more memory at the same price later but we really haven't gotten there yet-- gpu prices are still super infl…

For sure. Consumer GPU prices have fallen dramatically but they're still competitive enough; meanwhile last-gen (or later) no-output GPUs with passive cooling and single 8-pin CPU-type connectors are insanely cheap. P40s are readily available for $190, less if you lowball eBay or Facebook sellers at volume discounts.

You can even find some weird retired Cirrascale servers on eBay that provide 8 high-speed PCIe lanes through risers with Tesla-specific connectors on a motherboard with tons of RAM (and a terrible CPU) and multiple PSUs for fractions of their release cost.

It's a great time to be buying 3-5 year old ML equipment for small businesses and hobbyists. I wonder if the prices will ever go up? Not that I'm interested in speculating, but it's a small slice of the market that I'm participating in... though maybe in a year it won't be. I imagine a lot of small businesses will bring ML/DL stuff internal for at least development/testing.

Re: Running large language models like ChatGPT on a single GPU

#257

Post by celdon25 »

JoshCole wrote: post #242
celdon25 wrote: post #233

Earlier quoted context omitted.

> That publicity isn't causally connected to success is belied by the existence of the advertising industry. The argument was about publicity as a reward motivator, not publicity itself, as a causal relation to success. To phrase it plainly: Which first-time founders do you think Paul Graham or Keith Rabois would more likely fund: Those who aspire to solve a problem with the world that they care passionately about? O…

> Which first-time founders do you think Paul Graham or Keith Rabois would more likely fund: Those who aspire to solve a problem with the world that they care passionately about? It is worth reflecting on the fact that the founder of OpenAI has had the strongest possible endorsement from Paul Graham. He was claimed to be among the greats before his successes: Paul Graham put him among Steve Jobs and Elon Musk. When P…

I'll have to check out Bellman's work, thanks!

Re: Running large language models like ChatGPT on a single GPU

#258

Post by p1esk »

lostmsu wrote: post #249
cardine wrote: post #235

Earlier quoted context omitted.

> I suppose if I had a 7 digit budget I could get a better deal. We got our "deal" when buying just a single server and have since bought many more with the same provider. We didn't spend 7 figures all at once, we did it piece-meal over time. There is nothing stopping you from getting much better prices. > I'm actually surprised you have 100% inference utilization - customer load typically scales dynamically, so with…

> We got our "deal" when buying just a single server and have since bought many more with the same provider. We didn't spend 7 figures all at once, we did it piece-meal over time. There is nothing stopping you from getting much better prices. If it is as easy as you make it sound, why would you not just share the vendor name? I personally would love an 8xH100 machine for transformer experiments, but $100k+ pricing ma…

8xH100 machine is ~300k I’ve heard.

Re: Running large language models like ChatGPT on a single GPU

#259

Post by lostmsu »

p1esk wrote: post #258
lostmsu wrote: post #249

Earlier quoted context omitted.

> We got our "deal" when buying just a single server and have since bought many more with the same provider. We didn't spend 7 figures all at once, we did it piece-meal over time. There is nothing stopping you from getting much better prices. If it is as easy as you make it sound, why would you not just share the vendor name? I personally would love an 8xH100 machine for transformer experiments, but $100k+ pricing ma…

8xH100 machine is ~300k I’ve heard.

Well, the person above claims 8xA100 significantly under $130k. I am curious to hear more.

Re: Running large language models like ChatGPT on a single GPU

#260

Post by JoshCole »

celdon25 wrote: post #257
JoshCole wrote: post #242

Earlier quoted context omitted.

> Which first-time founders do you think Paul Graham or Keith Rabois would more likely fund: Those who aspire to solve a problem with the world that they care passionately about? It is worth reflecting on the fact that the founder of OpenAI has had the strongest possible endorsement from Paul Graham. He was claimed to be among the greats before his successes: Paul Graham put him among Steve Jobs and Elon Musk. When P…

I'll have to check out Bellman's work, thanks!

BTW, Bellman actually coined the term curse of dimensionality [1]; got that confused with combinatorial explosion since it is a synonyms in the contexts I typically encounter it [2].

[1]: https://en.wikipedia.org/wiki/Curse_of_dimensionality

[2]: https://en.wikipedia.org/wiki/Combinatorial_explosion

OpenAI has a pretty good introduction to the Bellman equations in their Spinning Up in RL lessons [3]. Sutton's work in Reinforcement Learning also talks about Bellman's work quite a bit. Though Bellman was actually studying what he called dynamic programming problems his work is now considered foundational in reinforcement learning.

[3]: https://spinningup.openai.com/en/latest/

Uh, and for the dual mode observations the person that brought that to my attention was Noam Brown, not Bellman or Norvig. If you haven't already checked out his work, I recommend it above both Norvig and Bellman. He has some great talks on Youtube and I consider it a shame they aren't more widely viewed [4].

[4]: https://www.youtube.com/watch?v=cn8Sld4xQjg

Post reply on HN