61–70 of 274 posts
I just tried to run the example in the README, using the OPT-30B model. It appeared to download 60GiB of model files, and then it attempted to read all of it into RAM. My laptop has "only" 32GiB of RAM so it just ran out of memory.
I have recently written a paper on understanding transformer learning via the lens of coinduction & Hopf algebra. https://arxiv.org/abs/2302.01834 The learning mechanism of transformer models was poorly understood however it turns out that a transformer is like a circuit with a feedback. I argue that autodiff can be replaced with what I call in the paper Hopf coherence which happens within the single layer as opposed…
what
I just tried to run the example in the README, using the OPT-30B model. It appeared to download 60GiB of model files, and then it attempted to read all of it into RAM. My laptop has "only" 32GiB of RAM so it just ran out of memory.
Earlier quoted context omitted.
I would hope publicity isn’t the motivation for doing it though.
What motivation would be sufficiently noble?
Any chance these work on CPUs with any acceptable performance? I have a 10-core 20-thread monster CPU, but didn't bother with a dedicated GPU because I can't control something as simple as its temperature. See the complicated procedure that only works with the large proprietary driver here: https://wiki.archlinux.org/title/NVIDIA/Tips_and_tricks#Over...
Nope. 20 cores in a CPU, 2000 in a GPU, with much much faster memory and an architecture designed to chew through data as fast as possible.
Earlier quoted context omitted.
I would hope publicity isn’t the motivation for doing it though.
What motivation would be sufficiently noble?
It's really interesting that these models are written in Python. Anyone know how much of a speed up using a faster language here would have? Maybe it's already off-loading a lot of the computation to C (I know many Python libraries do this), but I'd love to know.
Earlier quoted context omitted.
The OPT175b model is massive. A lot of that system ram probably holds model data.
Interesting, though apparently the OPT175B model is 350GB: > You will need at least 350GB GPU memory on your entire cluster to serve the OPT-175B model. For example, you can use 4 x AWS p3.16xlarge instances, which provide 4 (instance) x 8 (GPU/instance) x 16 (GB/GPU) = 512GB memory. https://alpa.ai/tutorials/opt_serving.html (Scroll down to the second "Note", not far from the top) I wonder what FlexGen is doing.. a…
Only in FP16. In the paper they use int4 quantization to reduce it to a quarter of that. In addition to the model weights, there's also a KV cache that takes up considerable amounts of memory, and they use int4 on that as well.
> I wonder what FlexGen is doing.. a naive guess is a mix of SSD and system memory.
That's correct, but other approaches have done this as well. What's "new" here seems to be the optimized data access pattern in combination with some other interesting techniques (prefetching, int4 quantization, CPU offload).
This is cool! But I wonder if it's economical using cloud hardware. The author claims 1.12 tokens/second on the 175B parameter model (arguably comparable to GPT-3 Davinci). That's about 100k tokens a day on the GCP machine the author used. Someone double check my numbers here, but given the Davinci base cost of $0.02 per 1k tokens and GCP cost for the hardware listed "NVIIDA T4 (16GB) instance on GCP with 208GB of DR…
You've made one huge mistake: Davinci's $0.02 is not just per 1k tokens generated but also context tokens consumed . So if you generate 50 tokens per request with 1k context, the price is actually 20 times as large at $0.40 per 1k tokens generated - much less palatable, costing 3 times as much as the cloud hosted version of this. And that's not even taking into account the gigantic markup cloud services have.
So it's not correct to only modify OpenAI's prices to account for the ratio of context tokens to output tokens. Both of them get multiplied by 20 (if that's what your ratio is).
Earlier quoted context omitted.
On the other hand, one techie with a few million dollars... And you could train something like GPT-3 for cheaper than a superbowl commercial. That would get you a lot of publicity.
You can do it once, but probably not every day.