This might be an interesting language model. However people care about ChatGPT entirely due to its quality, which this doesn’t demonstrate yet.
[deleted]
ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
21–30 of 146 posts
Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
#22THe RWKV model seems really cool. If you could get transformer-like performance with an RNN, the “hard coded” context length problem might go away. (That said, RNNs famously have infinite context in theory and very short context in reality.) Is there a primer for what RWKV does differently? According to the Github page it seems the key is multiple channels of state with different decaying rates, giving I assume, a co…
> RNNs famously have infinite context in theory and very short context in reality.
any sources to read more about this please? its the first ive heard of it
Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
#23The readme does not seem to be geared towards people not familiar with the topic. My questions: - Is this on the run on consumer GPU scale, or run on 8 A100 scale or you can’t run it yourself ever scale? - How does it compare to other language models in quality/abilities? - What is the training data?
The large model weights are 14B, so at 16 bits per weight, it won't quite fit on one 3090 or 4090.
Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
#24Turns out it does not matter if you have transformer/MLP/lstm or whatever, as long as there are enough parameters and training epochs over large dataset things "just work"
Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
#25The readme does not seem to be geared towards people not familiar with the topic. My questions: - Is this on the run on consumer GPU scale, or run on 8 A100 scale or you can’t run it yourself ever scale? - How does it compare to other language models in quality/abilities? - What is the training data?
It does say on there they are training it on the Pile training data. And they have this bit comparing inference with GPT2-XL:
RWKV-3 1.5B on A40 (tf32) = always 0.015 sec/token, tested using simple pytorch code (no CUDA), GPU utilization 45%, VRAM 7823M
GPT2-XL 1.3B on A40 (tf32) = 0.032 sec/token (for ctxlen 1000), tested using HF, GPU utilization 45% too (interesting), VRAM 9655M
So it looks about twice as fast for inference while using only about 80% as much VRAM. Obviously at such a small size, just 1.5B, you can run it even on consumer GPUs but you could do that with GPT2 as well. If it remains 80% of VRAM usage when scaled up, we’re still talking 282GB once it’s the size of BLOOM w/ 176B parameters. So yeah still 8x A100 40GB cards I guess. Not going to be the Stable Diffusion of LLMs.
Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
#26[flagged]
[deleted]
Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
#27Turns out it does not matter if you have transformer/MLP/lstm or whatever, as long as there are enough parameters and training epochs over large dataset things "just work"
To some degree, because we keep recreating the truly essential components the crude "Turing machine completeness" way. In time as we analyze the resulting models, we may find what patterns emerge and optimize for them. The result will be smaller, faster models that perform like larger slower ones.
Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
#28Earlier quoted context omitted.
The leap in public exposure wasn't so much GPT3 to GPT3.5, it was attaching a clean UI to the model, (with sane defaults) and allowing people to talk to it like a person. Suddenly it became something 'real' then. (This is purely talking about the public popularity of GPT)
Well yes, having no context memory, being slightly worse and requiring either a monster rig to run or paying per prompt made it completely and utterly irrelevant. Even now that it's improved and free to use its actual practical usability is marginal at best given the rate of blatantly wrong info being spewed with 105% confidence at the moment.
> blatantly wrong info being spewed with 105% confidence
There are some approaches. For example in this paper they say truth has a certain logical consistency that is lacking in hallucinations and deception. So they find this latent direction that indicates truth in a frozen LLM. This actually works better than asking the model to self evaluate by text generation, or training with RLHF.
"Discovering Latent Knowledge in Language Models Without Supervision" https://arxiv.org/abs/2212.03827
There's also a video with the first author: "Making LLMs Say The Truth" https://www.youtube.com/watch?v=XSQ495wpWXs&t=1515s
Btw, I think this is one of the deepest discussions about LLM hallucinations and alignment I ever saw. Worth a watch, even if it is a bit long. Not every day something like this comes long.
Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
#29THe RWKV model seems really cool. If you could get transformer-like performance with an RNN, the “hard coded” context length problem might go away. (That said, RNNs famously have infinite context in theory and very short context in reality.) Is there a primer for what RWKV does differently? According to the Github page it seems the key is multiple channels of state with different decaying rates, giving I assume, a co…
> RNNs famously have infinite context in theory and very short context in reality. any sources to read more about this please? its the first ive heard of it
I am not sure this article will answer your question, but Karpathy has an article about RNNs.
Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
#30This might be an interesting language model. However people care about ChatGPT entirely due to its quality, which this doesn’t demonstrate yet.
The leap in public exposure wasn't so much GPT3 to GPT3.5, it was attaching a clean UI to the model, (with sane defaults) and allowing people to talk to it like a person. Suddenly it became something 'real' then. (This is purely talking about the public popularity of GPT)
I dunno, I played a lot of AI Dungeon when it was GPT3 powered. It was quite nonsencial sometimes. ChatGPT is much better due to reinforcement learning and optimisation they did with it