Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

github.com

Post reply on HN
Print view View original

121–130 of 146 posts

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#121

Post by PartiallyTyped »

lacasito25 wrote: post #100
Towaway69 wrote: post #99

Earlier quoted context omitted.

From guide linked above: > It is remarkable that such large multi-lingual model is openly available for everybody. Am I the only one thinking that this remark is a insight into societal failure? The model has been trained on global freely available content, anyone who has published on the Web has contributed. Yet the wisdom gained from our collective knowledge is assumed to be withheld from us. As the original remark…

How much money you think gpt3 training costed?

I recommend reading the first few chapters of "The conquest of bread".

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#122

Post by ornornor »

capableweb wrote: post #118
ornornor wrote: post #103

Earlier quoted context omitted.

I think it’s similar to how search engines keep their ranking formulas secret, and you can’t run your own off a copy of their index. Yet we also all contributed to it by publishing (and feeding it, for instance by following googles requirements for micro data). But we don’t own any of it.

Main difference with a search engine is that a search engine ultimately links back to you. So the user, interested in more or want to know where it comes from, ends up on your website. The same is not true for these AI tools. The output could have been contributed by you, someone else, or everyone, or a combination of those, but it'll never be clear who actually contributed and there will be no credit to anyone besid…

Didn’t think of it this way, that makes sense. Thank you

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#123

Post by PartiallyTyped »

nl wrote: post #36
rkwasny wrote: post #24

Turns out it does not matter if you have transformer/MLP/lstm or whatever, as long as there are enough parameters and training epochs over large dataset things "just work"

This isn't true - the model architecture matters a lot. In general RNNs cannot handle long term dependencies (ie, long pieces of text) because the gradient vanishes. It's unclear how this solves this problem although they do reference the "attention free transformer" paper: https://arxiv.org/abs/2105.14103

The key component is the linear attention[1] and residual connections.

[1] https://arxiv.org/abs/2006.16236

> Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from (N2) to (N), where N is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our linear transformers achieve similar performance to vanilla transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#124

Post by blagie »

JellyBeanThief wrote: post #89
Dylan16807 wrote: post #83

Earlier quoted context omitted.

For weights, the order of magnitude is the important part. And the sign bit. So you can get pretty good coverage with only 16 values.

Down that far, I start to wonder if trinary circuits might become useful again. fp4 with 1-3-0 would mean 27 values if the first bit were interpreted as binary. But--and an engineer should check me on this cause to me a transistor is a distant abstraction--I think you could double that to 54 values if you were clever with the sign bit and arithmetic circuitry. Maybe push it to 42 if only some of my intuition is wrong…

You're wrong on many levels.

The basic reason for binary is because it's generally faster, especially as you scale to smaller transistors with more noise.

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#125

Post by blagie »

wokwokwok wrote: post #86
corobo wrote: post #70

Earlier quoted context omitted.

How badly is bad? What sort of output are we talking? I am asking as I once had a Markov-chain IRC bot* and while it often struggled to string together a sentence, it was quite hilarious sometimes. Absolutely pointless other than the occasional laugh. Can it form sentences or are those small models completely unusable for anything? I'm not thinking OpenAI level uses - sort of compare a Postgres cluster to a SQLite fi…

> or are those small models completely unusable for anything? Sadly, they really offer almost no value. For the effort, you’re better off with an NLP framework like spacy. You can play with the small neo gpt models on hugging face, eg. https://huggingface.co/EleutherAI/gpt-neo-125M …but, the tldr is they’re cute to play with, but practically, the content they can generate is short, inconsistent and full of errors.

.... which is actually not of "almost no value." The value of smaller models is different. For example, I have anonymized data, with fields removed. The smaller models do fine for filling those fields in with plausible values.

The smaller models do okay for zero-shot clustering of data in many cases (e.g. liberal versus conservative text), and if not, with minimal training. For generating statistics or probabilistic information about large numbers of text, they're great.

GPT-3, they're not, but I use them in my day-to-day work quite a bit more than I thought I would. I bought a GPU for one purpose, and I find I spin it up a lot these days.

I /really/ want to be able to use a large-scale language model locally, though. For the types of things I'd like it for, such as helping me draft emails, I don't trust OpenAI with my data.

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#126

Post by blagie »

visarga wrote: post #114
wokwokwok wrote: post #52

Earlier quoted context omitted.

You details of what is required can be found per project. eg. https://github.com/fauxpilot/fauxpilot/blob/main/setup.sh#L5... /shrug If you don't want to run it locally, don't. If the hardware you have doesn't support it, don't. ...but to say it's not possible is, again, bluntly, wrong.

You need something like an NVIDIA DGX station which tops $300K and uses over 5kW of power. https://www.deltacomputer.com/nvidia-dgx-h100-640gb.html

No, you can get away with under $20k these days. The models can now distribute adequately well, so you can get away with a swarm of thirty two 12GB 3060 Ti cards. 32*$400 = $12,800. Add in computers to stick them in and interlink, and that's sort of where you land.

When you needed this in one machine, the cost to build a box capable of running OPT was around $50-$150k. An 80GB GPU runs around $10k-$20k, and you need five of them.

So you can get away with a good bit less than an NVidia DGX Station. Unless you're a FAANG SWE of a high level of some kind, it's well outside of hobbyist use, but it's not an unreasonable expense for most business use. The way I think of this expense is it's 10% of the annual cost of a SWE. If it improves productivity by more than 10%, it's worth it for a one-person team. With a 10-person team, it's worth it at a 1% improvement.

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#127

Post by [deleted] »

sillysaurusx wrote: post #105
zaptrem wrote: post #95

Earlier quoted context omitted.

You might also mess with your loss function to force it to "make up its mind" as right now the blurry mess likely minimizes the error from the actual frame (which isn't really want you want).

Exactly! That was the exact thing I was trying to think of a way to do. Got any ideas? There’s discriminators, but after reading over prior work, it seems like they help, but they weren’t really a groundbreaking / effective solution. I had two harebrained ideas in mind. One is to add yolo style object detection. The difference between a blurry mess and a recognizable object is the fact that it’s a recognizable object…

[deleted]

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#128

Post by laughingman2 »

sillysaurusx wrote: post #105
zaptrem wrote: post #95

Earlier quoted context omitted.

You might also mess with your loss function to force it to "make up its mind" as right now the blurry mess likely minimizes the error from the actual frame (which isn't really want you want).

Exactly! That was the exact thing I was trying to think of a way to do. Got any ideas? There’s discriminators, but after reading over prior work, it seems like they help, but they weren’t really a groundbreaking / effective solution. I had two harebrained ideas in mind. One is to add yolo style object detection. The difference between a blurry mess and a recognizable object is the fact that it’s a recognizable object…

Have you checked RSSM approach in DreamerV1,V2,V3,PlaNet? It uses deterministic (GRU hidden state) and discrete stochastic latent states. The deterministic and stochastic (sampled) latent state are used to predict the next state. I think the stochastic state might help with your problem a bit.

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#129

Post by spyder »

logicallee wrote: post #106
johnc1 wrote: post #69

Earlier quoted context omitted.

> Can these models feasibly be run locally? Actually you can, it even works without GPU, here's a guide on running BLOOM (the open-source GPT-3 competitor of similar size) locally: https://towardsdatascience.com/run-bloom-the-largest-open-ac... The problem is performance: - if you have GPUs with > 330GB VRAM, it'll run fast - otherwise, you'll run from RAM or NVMe, but very slowly - generating one token every few min…

> if you have GPUs with > 330GB VRAM, it'll run fast What kind of GPU's have that that are available to consumers, how much would such a kit cost roughly?

He means multiple GPUs in parallel that have a combined VRAM of that size. So around 4 x NVIDIA A100 80GB, which you can get for around $8.4 / hour in the cloud. or 7 x NVIDIA A6000 or A40 48GB for $5.5 / hour

So not exactly cheap or easy yet for the everyday user, but I believe the models will become smaller and more affordable to run, these are just the "first" big research models focused demonstrating some usefulness after that they can be more focus on the size and speed optimizations. There are multiple methods and lot of research into making them smaller with distilling them, converting to lower precision, pruning the less useful weights, sparsifying. Some achieve around 40% size reduction 60% speed improvement with minimal accuracy loss, others achieve 90% sparsity. So there is hope to run them or similar models on a single but powerful computer.

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#130

Post by Hendrikto »

solomatov wrote: post #50
aljungberg wrote: post #20

THe RWKV model seems really cool. If you could get transformer-like performance with an RNN, the “hard coded” context length problem might go away. (That said, RNNs famously have infinite context in theory and very short context in reality.) Is there a primer for what RWKV does differently? According to the Github page it seems the key is multiple channels of state with different decaying rates, giving I assume, a co…

My understanding is that RNNs aren't worse than Transformers per se, they are just slower to train, and use GPU much more efficiently, i.e. much more stuff could be run in parallel.

Also slower to perform inference on. RNNs have to be much more sequential.
Post reply on HN