131–140 of 274 posts
Earlier quoted context omitted.
On the other hand, one techie with a few million dollars... And you could train something like GPT-3 for cheaper than a superbowl commercial. That would get you a lot of publicity.
Is there information out there about how much it cost (in time or human-hours) to do the additional training necessary to make chatGPT? I am genuinely curious what the scale of the effort was.
If this works well, it will be a game changer. Requiring a fleet of $10k+ GPUs will kill any hope of wide spread adoption of open source "competitors" to GPT-3. Stable Diffusion is so popular because it can run on hardware mere mortals can own.
As a former AI accelerator employee (laid off), I'm kind of happy I was laid off because I realistically don't see a need for specialized hardware anymore. Large companies can afford Nvidia. Nvidia's software stack is best in class. There's no business need here and the model execution is increasingly becoming possible on single consumer GPUs. The only place where I see specialized chips excelling is on the edge or i…
Earlier quoted context omitted.
This is most likely aimed at people running models locally. And a homelab with 3090s/4090s is one or two orders of magnitude cheaper than GCP, if you use them continuously.
I do not know anyone offhand with a 200+GB RAM home computer. The GPU is not all that is needed; you need to keep the parameters and other stuff in memory too.
I don't think 200gb ram is actually required, that's just what that cloud vm was spec'd with. Though the 175b model should see benefit with ram even beyond 200gb.
Got the ops-6.7b chatbot running on a windows machine with a 3090 in mere minutes. The only difference was to install the cuda pytorch `pip install torch==1.13.1+cu117 --extra-index-url https://download.pytorch.org/whl/cu117 ` just like in stable diffusion's case. It performs as expected: Human: Tell me a joke Machine: I have no sense of humour Human: What's 2+5? Machine: I cannot answer that.
6.7b is pretty small, no? Do you even need offloading for that on a 3090? I'd be curious to see what's needed to run opt-30b or opt-66b with reasonable performance. The README suggests that even opt-175b should be doable with okay performance on a single NVIDIA T4 if you have enough RAM.
Earlier quoted context omitted.
> I couldn't figure out if I was supposed to be chasing down Apple Metal or OpenCL backends. If you want cross-platform compatibility (kinda), go for OpenCL, if you want the best performance go for Metal. Both use a very similar language for kernels, but Metal is generally more efficient. > Have you had any luck? Not in ML, but I'm doing a lot of GPGPU on Metal, I recently started doing it in Rust. A bit less conveni…
I'm not familiar with Metal, but on Apple Silicon aren't CPU and GPU memory completely shared?
I looked it up and turns out OpenCL also supports zero-copy buffers, so I edited my comment accordingly!
If this works well, it will be a game changer. Requiring a fleet of $10k+ GPUs will kill any hope of wide spread adoption of open source "competitors" to GPT-3. Stable Diffusion is so popular because it can run on hardware mere mortals can own.
do/did you seriously think that the processing requirement was going to “kill” that possibility? the history of computing clearly indicates that either the requirements for running or the bar for owning this technology was always going to drop
Earlier quoted context omitted.
This may be a flawed approach, but an interesting idea would be to use the current models as a preprocessor to generate a huge "labelled" dataset of inputs and outputs, and then using that more accurate and specific data to train a smaller one that would fit. It likely wouldn't have nearly as much general knowledge since the data just wouldn't be there but the behaviour could be similar?
Look up knowledge distillation
Earlier quoted context omitted.
While OPT-175B is great to have publicly available, it needs a lot more training to achieve good results. Meta trained OPT on 180B tokens, compared to 300B that GPT-3 saw. And the Chinchilla scaling laws suggest that almost 4T tokens would be required to get the most bang for compute buck. And on top of that, there are some questions on the quality of open source data (The Pile) vs OpenAI’s proprietary dataset, which…
It’s fun to think about a few billion weights being the difference between useless and gold.
Earlier quoted context omitted.
How much system RAM are you running with? And I'm guessing it wouldn't hurt to have a fast SSD for disk offloading?
128GB, but by turning on compression I managed to fit the whole thing on the GPU. I did try it off a mix of RAM and SSD as well, and it was slower but still usable. Presumably disk speed matters a lot.
Any chance these work on CPUs with any acceptable performance? I have a 10-core 20-thread monster CPU, but didn't bother with a dedicated GPU because I can't control something as simple as its temperature. See the complicated procedure that only works with the large proprietary driver here: https://wiki.archlinux.org/title/NVIDIA/Tips_and_tricks#Over...
I don't know about these large models but I saw on a random HN comment earlier in a different topic where someone showed a GPT-J model on CPU only: https://github.com/ggerganov/ggml I tested it on my Linux and Macbook M1 Air and it generates tokens at a reasonable speed using CPU only. I noticed it doesn't quite use all my available CPU cores so it may be leaving some performance on the table, not sure though. The GP…