Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

github.com

Post reply on HN
Print view View original

91–100 of 146 posts

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#91

Post by tempay »

Dylan16807 wrote: post #87
johnc1 wrote: post #69

Earlier quoted context omitted.

> Can these models feasibly be run locally? Actually you can, it even works without GPU, here's a guide on running BLOOM (the open-source GPT-3 competitor of similar size) locally: https://towardsdatascience.com/run-bloom-the-largest-open-ac... The problem is performance: - if you have GPUs with > 330GB VRAM, it'll run fast - otherwise, you'll run from RAM or NVMe, but very slowly - generating one token every few min…

If it fits in system memory, is it still faster on GPU than CPU? Does that involve swapping out one layer at a time? Otherwise I'm very curious how it handles the PCIe latency. Enough system memory to fit 84GB isn't all that expensive...

Yes, the connection between system memory and the GPU isn’t fast enough to keep the compute units fed with data to process. Generally PCIe latency isn’t as much of a problem as bandwidth.

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#92

Post by jszymborski »

solomatov wrote: post #53
swyx wrote: post #22

Earlier quoted context omitted.

> RNNs famously have infinite context in theory and very short context in reality. any sources to read more about this please? its the first ive heard of it

Naive RNN have vanishing gradient, but LSTMs and GRUs are much better in this respect.

While this is true, and was a major advantage of LSTMs/GRUs, they still suffer from vanishing gradients.

w.r.t proteins, our sequences often surpass 1500 amino acids and that is really tough for an LSTM to stably train on.

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#93

Post by ggerganov »

corobo wrote: post #70
wokwokwok wrote: post #42

Earlier quoted context omitted.

> Can these models feasibly be run locally? Bluntly, no. The models which are small enough to run locally perform so badly it’s not worth bothering. To run inference on the large models the perform decently you need the equivalent of two or three top end graphics cards. If you're serious about looking into it now, consider looking at this project that lets you run a bunch of independent machines as a cluster for infe…

How badly is bad? What sort of output are we talking? I am asking as I once had a Markov-chain IRC bot* and while it often struggled to string together a sentence, it was quite hilarious sometimes. Absolutely pointless other than the occasional laugh. Can it form sentences or are those small models completely unusable for anything? I'm not thinking OpenAI level uses - sort of compare a Postgres cluster to a SQLite fi…

I was recently playing with the GPT-2 and GPT-J models. Results are often non-sensical for any practical purposes, but I think can be used for making something fun - similar to your IRC bot idea.

If you are interested in running these models yourself without having a beefy GPU, you can try my custom inference implementation. It's in pure C/C++ without any 3rd party dependencies, runs straight on the CPU and builds very easily. I think it is relatively well optimised. For example, on a MacBook M1 Pro I can run GPT-2 XL (1.5B params) at 42ms/token and GPT-J / GPT-JT (6B params) at 125ms/token.

Here are a couple of generated examples using GPT-J:

https://github.com/ggerganov/ggml/tree/master/examples/gpt-j

These are examples using zero-shot prompt where the model auto-completes a text given a starting prompt. You can try to make a conversation bot with a few-shot prompt, but it's not great. Probably the model needs some fine-tuning for that to become feasible.

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#94

Post by dragonwriter »

anigbrowl wrote: post #73
wokwokwok wrote: post #42

Earlier quoted context omitted.

> Can these models feasibly be run locally? Bluntly, no. The models which are small enough to run locally perform so badly it’s not worth bothering. To run inference on the large models the perform decently you need the equivalent of two or three top end graphics cards. If you're serious about looking into it now, consider looking at this project that lets you run a bunch of independent machines as a cluster for infe…

How much of this is the language vs the vast amount of passably accurate domain knowledge? ChatGPT etc. seem magic because they can answer questions about virtually anything with a high degree of plausibility. It often gets specific facts wrong, but the general contours are correct. Many of us know a lot of trivia/specialist knowledge, but I don't think anyone is as broadly informed as ChatGPT appears to be. It's not…

> How much of this is the language vs the vast amount of passably accurate domain knowledge?

LLMs don’t have domain knowledge, its all language.

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#95

Post by zaptrem »

sillysaurusx wrote: post #35
swyx wrote: post #31

Earlier quoted context omitted.

it doesnt touch on the "infinite context in theory and very short context in reality" piece which is what i was asking about

I can confirm it from what we’re seeing on a video prediction task. Future frames end up blurry. The first frame is sharp, but by frame 3 it’s only crisp when it’s very certain of its prediction. Any kind of rare movement, it goes “I kinda know what it roughly looks like” and smears fingerpaint all over the canvas. The overall trajectory looks ok, so I’ll be more rigorously investigating whether it’s possible to sque…

You might also mess with your loss function to force it to "make up its mind" as right now the blurry mess likely minimizes the error from the actual frame (which isn't really want you want).

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#96

Post by zaptrem »

haint_ wrote: post #84

From the provided example: Q: How would I make for loop in python? A: I can help you create an AI chat bot. It would talk to you like a human. (additional text that is not relevant to the prompt) It is just me or this does not seem right?

This is only a 1.5b parameter model. This is in line with that. GPT3.5 is ~175b params.

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#97

Post by Dylan16807 »

JellyBeanThief wrote: post #89
Dylan16807 wrote: post #83

Earlier quoted context omitted.

For weights, the order of magnitude is the important part. And the sign bit. So you can get pretty good coverage with only 16 values.

Down that far, I start to wonder if trinary circuits might become useful again. fp4 with 1-3-0 would mean 27 values if the first bit were interpreted as binary. But--and an engineer should check me on this cause to me a transistor is a distant abstraction--I think you could double that to 54 values if you were clever with the sign bit and arithmetic circuitry. Maybe push it to 42 if only some of my intuition is wrong…

But why? There's nothing special about having 4 storage elements. If you want 54 values then 6 bits are going to be just as effective as 4 trits, and easier to implement in every way.

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#98

Post by afro88 »

adam_arthur wrote: post #78
johnc1 wrote: post #69

Earlier quoted context omitted.

> Can these models feasibly be run locally? Actually you can, it even works without GPU, here's a guide on running BLOOM (the open-source GPT-3 competitor of similar size) locally: https://towardsdatascience.com/run-bloom-the-largest-open-ac... The problem is performance: - if you have GPUs with > 330GB VRAM, it'll run fast - otherwise, you'll run from RAM or NVMe, but very slowly - generating one token every few min…

Pretty cool! Honestly even if it were to take a few minutes per response, that's likely sufficient for many use cases. I'd get value out of that if it allowed bypassing a paywall. I'm curious how these models end up being monetized/supported financially, as they sound expensive to run at scale. The required disk space seems the biggest barrier for local.

If it's a few minutes per token you might be waiting a lot longer for a full response: https://blog.quickchat.ai/post/tokens-entropy-question/

I also wonder how open.ai etc provides access to these for free. Reminds me of the adage from when Facebook rose to popularity: "if something is free, 'you' are the product". Perhaps to gather lots more conversational training data for fine tuning.

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#99

Post by Towaway69 »

johnc1 wrote: post #69
adam_arthur wrote: post #38

So what's the level of effort to create ChatGPT equivalent products? Is it something where we'll have 100s of competing AIs, or is it gated to only a few large companies? Not up to date on current training/querying costs. Can these models feasibly be run locally? Given the large number of competitors already announced to ChatGPT, I fail to see how the space will be easily defensible or monetizable (despite large valu…

> Can these models feasibly be run locally? Actually you can, it even works without GPU, here's a guide on running BLOOM (the open-source GPT-3 competitor of similar size) locally: https://towardsdatascience.com/run-bloom-the-largest-open-ac... The problem is performance: - if you have GPUs with > 330GB VRAM, it'll run fast - otherwise, you'll run from RAM or NVMe, but very slowly - generating one token every few min…

From guide linked above:

> It is remarkable that such large multi-lingual model is openly available for everybody.

Am I the only one thinking that this remark is a insight into societal failure? The model has been trained on global freely available content, anyone who has published on the Web has contributed.

Yet the wisdom gained from our collective knowledge is assumed to be withheld from us. As the original remark was one of surprise, the authors (and our) assumption is that trained models are expected to be kept from us.

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#100

Post by lacasito25 »

Towaway69 wrote: post #99
johnc1 wrote: post #69

Earlier quoted context omitted.

> Can these models feasibly be run locally? Actually you can, it even works without GPU, here's a guide on running BLOOM (the open-source GPT-3 competitor of similar size) locally: https://towardsdatascience.com/run-bloom-the-largest-open-ac... The problem is performance: - if you have GPUs with > 330GB VRAM, it'll run fast - otherwise, you'll run from RAM or NVMe, but very slowly - generating one token every few min…

From guide linked above: > It is remarkable that such large multi-lingual model is openly available for everybody. Am I the only one thinking that this remark is a insight into societal failure? The model has been trained on global freely available content, anyone who has published on the Web has contributed. Yet the wisdom gained from our collective knowledge is assumed to be withheld from us. As the original remark…

How much money you think gpt3 training costed?
Post reply on HN