171–180 of 274 posts
Earlier quoted context omitted.
This is most likely aimed at people running models locally. And a homelab with 3090s/4090s is one or two orders of magnitude cheaper than GCP, if you use them continuously.
I do not know anyone offhand with a 200+GB RAM home computer. The GPU is not all that is needed; you need to keep the parameters and other stuff in memory too.
Earlier quoted context omitted.
This is not true - the break even period is closer to 6-7 months.
A single 8xA100 server is ~150k. On demand cost to rent it is $8.8/hour. Do the math and don't forget the energy costs.
Earlier quoted context omitted.
A single 8xA100 server is ~150k. On demand cost to rent it is $8.8/hour. Do the math and don't forget the energy costs.
I'd suggest finding a cheaper vendor if that is the lowest price you can get for an 8xA100 server. We spend a lot on both and colo our servers so I've definitely done the math!
Keep in mind, there are three important advantages of cloud:
1. You only pay for what you use (hourly). What is utilization of your on-prem servers?
2. You don't have to pay upfront - easier to ask for budget
3. You can upgrade your hardware easily as soon as new GPU models become available.
If this works well, it will be a game changer. Requiring a fleet of $10k+ GPUs will kill any hope of wide spread adoption of open source "competitors" to GPT-3. Stable Diffusion is so popular because it can run on hardware mere mortals can own.
This will only happen if "Open"AI or other big orgs release the model weights, which only Stable Diffusion did. Cost to train is still astronomical.
Earlier quoted context omitted.
Meta has released the model weights for OPT-175B, which is used in the paper. There's also a lot of full release LLMs from other labs on the way as well.
Are there any that perform anywhere close to GPT-3?
Got the ops-30b chatbot running on 3090 24GB. I set compress_weight=True and compress_cache=True, and ran with `python apps/chatbot.py --model facebook/opt-30b --percent 100 0 100 0 100 0`. I also modified the prompt a bit to make it more... uh alive: Assistant: Did you know that Saturn is 97 times the size of Earth? Human: Are you sure? Assistant: What difference does size make, really, anyway? Human: You didn't ans…
This is amazing. Reminds me of claptrap from Borderlands
What will be the first game with ChatGPT integrated into the NPC dialog interactions?
My vote is Hitman, with variable voices....
Earlier quoted context omitted.
What's the advantage of purchasing a T4 instead of a 3090 or 4090?
Power consumption. A Tesla T4 with 16GB RAM will consume a mere 70W. An RTX 3090 will need at least 300W, and the Titan models go up to 450W.
If this works well, it will be a game changer. Requiring a fleet of $10k+ GPUs will kill any hope of wide spread adoption of open source "competitors" to GPT-3. Stable Diffusion is so popular because it can run on hardware mere mortals can own.
This will only happen if "Open"AI or other big orgs release the model weights, which only Stable Diffusion did. Cost to train is still astronomical.
I'm aware of one very credible one that has applied for a TPU grant from Google worth under $200K to train the whole model.
I think laion.ai will probably get their first with their fleet of A100s though.
> The batch size is tuned to a value that maximizes the generation throughput for each system.
> FlexGen cannot achieve its best throughput in [...] single-batch case.
For 175B models, this likely means that the system takes a few seconds for each generation step, but you can generate multiple sequences in parallel and get a good performance _per token_.
However, what you actually need for ChatGPT and interactive LM apps is to generate _one_ sequence reasonably quickly (so it takes The authors don't report the speed of the single-batch generation in the repo and the paper.