[flagged]
ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
41–50 of 146 posts
Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
#42So what's the level of effort to create ChatGPT equivalent products? Is it something where we'll have 100s of competing AIs, or is it gated to only a few large companies? Not up to date on current training/querying costs. Can these models feasibly be run locally? Given the large number of competitors already announced to ChatGPT, I fail to see how the space will be easily defensible or monetizable (despite large valu…
> Can these models feasibly be run locally?
Bluntly, no.
The models which are small enough to run locally perform so badly it’s not worth bothering.
To run inference on the large models the perform decently you need the equivalent of two or three top end graphics cards.
If you're serious about looking into it now, consider looking at this project that lets you run a bunch of independent machines as a cluster for inference using Bloom:
https://github.com/bigscience-workshop/petals/wiki/Launch-yo...
(You'll need around 200GB of GPU memory across the machines in the swarm)
Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
#43So what's the level of effort to create ChatGPT equivalent products? Is it something where we'll have 100s of competing AIs, or is it gated to only a few large companies? Not up to date on current training/querying costs. Can these models feasibly be run locally? Given the large number of competitors already announced to ChatGPT, I fail to see how the space will be easily defensible or monetizable (despite large valu…
> Can these models feasibly be run locally? Bluntly, no. The models which are small enough to run locally perform so badly it’s not worth bothering. To run inference on the large models the perform decently you need the equivalent of two or three top end graphics cards. If you're serious about looking into it now, consider looking at this project that lets you run a bunch of independent machines as a cluster for infe…
Two or three top GPUs? Thats basically nothing for a professinal project or even an investeded hobby
Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
#44The readme does not seem to be geared towards people not familiar with the topic. My questions: - Is this on the run on consumer GPU scale, or run on 8 A100 scale or you can’t run it yourself ever scale? - How does it compare to other language models in quality/abilities? - What is the training data?
It does say on there they are training it on the Pile training data. And they have this bit comparing inference with GPT2-XL: RWKV-3 1.5B on A40 (tf32) = always 0.015 sec/token, tested using simple pytorch code (no CUDA), GPU utilization 45%, VRAM 7823M GPT2-XL 1.3B on A40 (tf32) = 0.032 sec/token (for ctxlen 1000), tested using HF, GPU utilization 45% too (interesting), VRAM 9655M So it looks about twice as fast for…
I'm pretty sure those numbers are for training, not inference. I've run it on _CPU_ and gotten ~1 token per second.
Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
#45Earlier quoted context omitted.
> Can these models feasibly be run locally? Bluntly, no. The models which are small enough to run locally perform so badly it’s not worth bothering. To run inference on the large models the perform decently you need the equivalent of two or three top end graphics cards. If you're serious about looking into it now, consider looking at this project that lets you run a bunch of independent machines as a cluster for infe…
Two or three top GPUs? Thats basically nothing for a professinal project or even an investeded hobby
[flagged]
Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
#46THe RWKV model seems really cool. If you could get transformer-like performance with an RNN, the “hard coded” context length problem might go away. (That said, RNNs famously have infinite context in theory and very short context in reality.) Is there a primer for what RWKV does differently? According to the Github page it seems the key is multiple channels of state with different decaying rates, giving I assume, a co…
There's already research that tries to fix this problem with transformers in general, like Transformer-XL [1]. I'm a bit puzzled that I don't see much interest in getting a pre-trained model out that uses this architecture---it seems to give good results. [1]: https://arxiv.org/abs/1901.02860
T5 uses relative positional encoding
Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
#47Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
#48Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
#49Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
#50THe RWKV model seems really cool. If you could get transformer-like performance with an RNN, the “hard coded” context length problem might go away. (That said, RNNs famously have infinite context in theory and very short context in reality.) Is there a primer for what RWKV does differently? According to the Github page it seems the key is multiple channels of state with different decaying rates, giving I assume, a co…
My understanding is that RNNs aren't worse than Transformers per se, they are just slower to train, and use GPU much more efficiently, i.e. much more stuff could be run in parallel.