Amazing,
i too think that the advent of "smaller" models will be the general release of pcie 5 nvme as caches!
Running large language models like ChatGPT on a single GPU
221–230 of 274 posts
Re: Running large language models like ChatGPT on a single GPU
#222IMO the title should be renamed to refer to that this is only for inference ("running" could imply training as well):
"Running inference for large language models like ChatGPT on a single GPU".
Re: Running large language models like ChatGPT on a single GPU
#223This space is gonna end up looking a lot like the compression space, there will be a few open source, publicly used AI’s that are pretty good for most people. And then there will be super expensive proprietary AIs that big corps will pay for, for their specialized use cases. The only people who will even know about those specialized AI’s existence will be the type of people who need them and everyone else in the world will think the best you can do is zip.
Re: Running large language models like ChatGPT on a single GPU
#224Earlier quoted context omitted.
Well maybe not every day, but having a short feedback loop and the ability to run your code multiple times with different variations is generally considered to be a prerequisite for software development. If you actually want to keep developing the model, you need the funding to be able to train it more than once.
> but having a short feedback loop and the ability to run your code multiple times with different variations is generally considered to be a prerequisite for software development This is not "software development" in general, this is LLM training. It's not like you're building some regular app, api, or backend.
If you are claiming that training a LLM literally only one time is enough and there is no need to train it more than once, you are wrong. The researchers who created OPT didn't go into a basement for 12 months, then come out, train their model once, hit publish, and go to coffee. That is a fantasy. Likewise, if a CS student wants to dabble in this research, they need the ability to train more than once.
I'm not gonna engage in a rhetorical argument about whether this should be called "software development" or "LLM development" or something else. That's unrelated to the question of how much training is required.
Re: Running large language models like ChatGPT on a single GPU
#225Any chance these work on CPUs with any acceptable performance? I have a 10-core 20-thread monster CPU, but didn't bother with a dedicated GPU because I can't control something as simple as its temperature. See the complicated procedure that only works with the large proprietary driver here: https://wiki.archlinux.org/title/NVIDIA/Tips_and_tricks#Over...
The other answer give you a few of the current solutions.
In the long term I am hoping that JAX (/XLA) will get better support for the CPU backend of their compiler and in particular, use SIMD and multicore better than it currently does.
It is very doable (just low priority) and it would mean that a lot of models could get close to optimal CPU performances out of the box which would be a step forward for accessibility.
Re: Running large language models like ChatGPT on a single GPU
#226Out of curiosity, why aren't we crowd sourcing distributed training of LLMs where anyone can join by bringing their hardware or data? Moreover find a way to incorporate this into a blockchain so there is full transparency but also add in differential privacy to protect every participant. Am I being too crazy here?
Just make sure it's written in Rust, uses a Sveltekit frontend and .
And SQLite as local cache with CRDTs enabled whereas everything else from text search to queuing on PostgreSQL?
Re: Running large language models like ChatGPT on a single GPU
#227Earlier quoted context omitted.
> but having a short feedback loop and the ability to run your code multiple times with different variations is generally considered to be a prerequisite for software development This is not "software development" in general, this is LLM training. It's not like you're building some regular app, api, or backend.
If you are claiming that training a LLM literally only one time is enough and there is no need to train it more than once, you are wrong. The researchers who created OPT didn't go into a basement for 12 months, then come out, train their model once, hit publish, and go to coffee. That is a fantasy. Likewise, if a CS student wants to dabble in this research, they need the ability to train more than once. I'm not gonna…
>If you are claiming that training a LLM literally only one time is enough and there is no need to train it more than once, you are wrong.
No, I'm rather claiming that what you claimed is wrong in the context of LLM training: "Well maybe not every day, but having a short feedback loop and the ability to run your code multiple times with different variations is generally considered to be a prerequisite for software development".
LLM training is not the same as writing a program and "running your code with different variations". For LLM you don't need to quickly rerun everything with some new corpus - it would be nice, but it's neither a prerequisite, not even crucial for any current use.
Hell, it's not even a "prerequisite" in programming, just good to have. Tons of great programs have been written with very slow build times, without quick edit/compile/build/run cycles.
Re: Running large language models like ChatGPT on a single GPU
#228Earlier quoted context omitted.
If you are claiming that training a LLM literally only one time is enough and there is no need to train it more than once, you are wrong. The researchers who created OPT didn't go into a basement for 12 months, then come out, train their model once, hit publish, and go to coffee. That is a fantasy. Likewise, if a CS student wants to dabble in this research, they need the ability to train more than once. I'm not gonna…
> If you are claiming that training a LLM literally only one time is enough and there is no need to train it more than once, you are wrong. No, I'm rather claiming that what you claimed is wrong in the context of LLM training: "Well maybe not every day, but having a short feedback loop and the ability to run your code multiple times with different variations is generally considered to be a prerequisite for software d…
I wasn't talking about running the same code with a new corpus. For that kind of use case one can simply fine tune the pretrained model. The example I gave was "if a CS student wants to dabble in this research".
You said "LLM training is not the same as writing a program and running your code with different variations". How do you think these LLMs were made, seriously? Do you think Facebook researchers sat down for 12 months and wrote code non stop without compiling it once, until the program was finished and was used to train the LLM literally only one time?
Re: Running large language models like ChatGPT on a single GPU
#229> ...a high-throughput generation engine for running large language models with limited GPU memory (e.g., a 16GB T4 GPU or a 24GB RTX3090 gaming card!).
laughs in 6 gb vram and no tensor cores.
Re: Running large language models like ChatGPT on a single GPU
#230A lot of people are looking at this wrong. A $350 3060Ti has 12GB RAM. If there's a way to run models locally, it opens up the door to: 1) Privacy-sensitive applications 2) Tinkering 3) Ignoring filters 4) Prototyping 5) Eventually, a bit of extra training The upside isn't so much cost / performance, as local control over a cloud-based solution.
Nitpicky but the RTX 3060 (non-Ti) has a variant with 12 GB, whilst the Ti is 8 GB. Agree with your points though