Wait until we have a game whose levels are procedural generated in all respects - where the geometry of the wold/landscape is procedural generated through the interactions you have with the AI NPCs whereby they generate the path - and there could be evil ones that direct you down a dark path when you piss them off... and oposite...
Running large language models like ChatGPT on a single GPU
191–200 of 274 posts
Re: Running large language models like ChatGPT on a single GPU
#192Re: Running large language models like ChatGPT on a single GPU
#193Note that the authors report the speed of generating many sequences in parallel (per token): > The batch size is tuned to a value that maximizes the generation throughput for each system. > FlexGen cannot achieve its best throughput in [...] single-batch case. For 175B models, this likely means that the system takes a few seconds for each generation step, but you can generate multiple sequences in parallel and get a…
I spoke with the authors of the paper; the leftmost points in Figure 1 were generated with batch-size 1, indicating ~1.2x and ~2x improvements in speed over DeepSpeed for 30B and 175B models respectively. For reference, this is speeding up from ~0.009tokens/s to about ~0.02tokens/s on 175B.
These results are generally unimpressive, of course. Most of the improvements at that point are attributable to the authors making use of a stripped down library for autoregressive sampling. HN falling for garbage once again...
Re: Running large language models like ChatGPT on a single GPU
#194Earlier quoted context omitted.
Meta has released the model weights for OPT-175B, which is used in the paper. There's also a lot of full release LLMs from other labs on the way as well.
While OPT-175B is great to have publicly available, it needs a lot more training to achieve good results. Meta trained OPT on 180B tokens, compared to 300B that GPT-3 saw. And the Chinchilla scaling laws suggest that almost 4T tokens would be required to get the most bang for compute buck. And on top of that, there are some questions on the quality of open source data (The Pile) vs OpenAI’s proprietary dataset, which…
OPT-175B isn't publicly available, sadly. It's available to research institutions, which is much better than "Open"AI, but it doesn't help us hobbyists/indie researchers much.
Re: Running large language models like ChatGPT on a single GPU
#195Earlier quoted context omitted.
No, Stable Diffusion isn't the only one to release their weights. OpenAI hasn't been releasing weights for ChatGPT, but Stable Diffusion isn't the only ones releasing weights [1]. [1]: https://huggingface.co/
yeah there an absolute pile of LLMs that are fully open-source. OpenAI’s GPT2 for one, but also Bloom, OPT, GPT-J, and I’m sure myriad others too
Nothing in the ballpark of GPT-3.5 or Prometheus yet, but we'll get there.
Re: Running large language models like ChatGPT on a single GPU
#196A lot of people are looking at this wrong. A $350 3060Ti has 12GB RAM. If there's a way to run models locally, it opens up the door to: 1) Privacy-sensitive applications 2) Tinkering 3) Ignoring filters 4) Prototyping 5) Eventually, a bit of extra training The upside isn't so much cost / performance, as local control over a cloud-based solution.
I have that exact card, this maybe the nudge where I remove windows from the computer and try out linux gaming (and local GPT)
I've had great results recently using Steam/Proton on Arch with my AMD 6750XT.
Re: Running large language models like ChatGPT on a single GPU
#197Earlier quoted context omitted.
At a very high level yes. There is also the very important step of efficiently laying out data in the GPU memory to compute tensor values in the kernels.
Can you confirm if OpenCL has been deprecated going forward for Apple Silicon please? Also, should I be able to expect to use OpenCL version 3.0 on Apple Silicon, or only v1.2 or 2.0 or something else?
Yes, according to Apple official documentation, OpenCL was deprecated as of macOS 10.14. It is reported to work, including on Apple Silicon (M1 and M2), but don't expect any updates.
[1]: https://developer.apple.com/library/archive/documentation/Pe...
Re: Running large language models like ChatGPT on a single GPU
#198Earlier quoted context omitted.
Why would you want to retrain it from scratch every day? Stable Diffusion doesn't do that either.
Well maybe not every day, but having a short feedback loop and the ability to run your code multiple times with different variations is generally considered to be a prerequisite for software development. If you actually want to keep developing the model, you need the funding to be able to train it more than once.
To summarize this discussion, we went from "this might mean we don't need a fleet of $10k+ GPUs to even run a LLM" to "yeah but an individual couldn't train one every day though". These goalposts are breaking the sound barrier.
Re: Running large language models like ChatGPT on a single GPU
#199Note that the authors report the speed of generating many sequences in parallel (per token): > The batch size is tuned to a value that maximizes the generation throughput for each system. > FlexGen cannot achieve its best throughput in [...] single-batch case. For 175B models, this likely means that the system takes a few seconds for each generation step, but you can generate multiple sequences in parallel and get a…
I spoke with the authors of the paper; the leftmost points in Figure 1 were generated with batch-size 1, indicating ~1.2x and ~2x improvements in speed over DeepSpeed for 30B and 175B models respectively. For reference, this is speeding up from ~0.009tokens/s to about ~0.02tokens/s on 175B. These results are generally unimpressive, of course. Most of the improvements at that point are attributable to the authors maki…
Calling this garbage is absolutely wild. The authors make it very clear that this is optimized for throughput and not latency. Throughput focused scenarios absolutely do exist, editorializing this as "running large language models like ChatGPT" and focusing on chatbot applications is the fault of HN.
It's also a neat result that fp4 quantization doesn't cause much issue even at 175b, though that kinda was to be expected.
Re: Running large language models like ChatGPT on a single GPU
#200I recently bought a T4 to go with my epyc 7402 and 512GB ram for fun and this looks like a great use case. Thanks!
What's the advantage of purchasing a T4 instead of a 3090 or 4090?
A lot of 2U cases won't fit a consumer GPU. Furthermore, Tesla-equivalents are usually either significantly cheaper than their consumer counterpart (for last-gen and older GPUs) or similar in price with far more RAM.
I bought a bunch of Tesla P40s at a really low price compared to what 1080tis are going for.