Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

github.com

Post reply on HN
Print view View original

61–70 of 146 posts

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#61

Post by adam_arthur »

throwifasd wrote: post #56
wokwokwok wrote: post #52

Earlier quoted context omitted.

You details of what is required can be found per project. eg. https://github.com/fauxpilot/fauxpilot/blob/main/setup.sh#L5... /shrug If you don't want to run it locally, don't. If the hardware you have doesn't support it, don't. ...but to say it's not possible is, again, bluntly, wrong.

yeah, that's not what op asked about. That's one of those smaller models that "suck" as you said so confidently in your first comment. Not even close to gpt3/chatgpt. Next time when I talk bullshit I just post a random py file to make my argument. lmao

Though if true, it does beg the question, how can these services possibly be offered profitably if a single query requires many high end GPUs to run?

Doesn't really track logically either. Though I'm not in the know. Perhaps ChatGPT is run with massive operating losses and its all VC subsidized.

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#62

Post by eternalban »

aljungberg wrote: post #20

THe RWKV model seems really cool. If you could get transformer-like performance with an RNN, the “hard coded” context length problem might go away. (That said, RNNs famously have infinite context in theory and very short context in reality.) Is there a primer for what RWKV does differently? According to the Github page it seems the key is multiple channels of state with different decaying rates, giving I assume, a co…

They explains it here:

https://github.com/BlinkDL/RWKV-LM#the-rwkv-language-model-a...

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#63

Post by throwifasd »

adam_arthur wrote: post #61
throwifasd wrote: post #56

Earlier quoted context omitted.

yeah, that's not what op asked about. That's one of those smaller models that "suck" as you said so confidently in your first comment. Not even close to gpt3/chatgpt. Next time when I talk bullshit I just post a random py file to make my argument. lmao

Though if true, it does beg the question, how can these services possibly be offered profitably if a single query requires many high end GPUs to run? Doesn't really track logically either. Though I'm not in the know. Perhaps ChatGPT is run with massive operating losses and its all VC subsidized.

This post [0] is a good primer and they are talking about a 20b model gpt3 has 175b. Their other posts go more into details but yes, it really is a massive operation.

Some hard facts from here [1] talking about BLOOM with 175b parameters..

>> Installing the full 175B version is a challenge though as it requires around 350GB of GPU VRAM, which is not something one can easily afford.

But hey why deploy a model like chatGPT at all when you guys can be confidently incorrect all by yourselves.

[0] https://nlpcloud.com/deploying-gpt-neox-20-production-focus-...

[1] https://nlpcloud.com/chatgpt-open-source-alternatives.html

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#65

Post by lolspace »

adam_arthur wrote: post #60
wokwokwok wrote: post #42

Earlier quoted context omitted.

> Can these models feasibly be run locally? Bluntly, no. The models which are small enough to run locally perform so badly it’s not worth bothering. To run inference on the large models the perform decently you need the equivalent of two or three top end graphics cards. If you're serious about looking into it now, consider looking at this project that lets you run a bunch of independent machines as a cluster for infe…

Hmmm, 2-3 high end GPUs implies it's likely not very far off from mainstream. Maybe runnable on the average device within 10-20 years... perhaps even sooner if the model/software can be optimized?

20 years?

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#66

Post by jacobr1 »

adam_arthur wrote: post #61
throwifasd wrote: post #56

Earlier quoted context omitted.

yeah, that's not what op asked about. That's one of those smaller models that "suck" as you said so confidently in your first comment. Not even close to gpt3/chatgpt. Next time when I talk bullshit I just post a random py file to make my argument. lmao

Though if true, it does beg the question, how can these services possibly be offered profitably if a single query requires many high end GPUs to run? Doesn't really track logically either. Though I'm not in the know. Perhaps ChatGPT is run with massive operating losses and its all VC subsidized.

One thing to consider, there is considerable asymmetry between the high training costs and the lower (relatively) operating costs.

> Perhaps ChatGPT is run with massive operating losses and its all VC subsidized.

I'm not in the know either, but the current access is providing both marketing awareness, and significant human training/feedback to be used for improvements on future commercial projects. It could certainly be the case that that benefits outweigh costs compared to the classic, dumber "get eyeballs now, monetize later" strategy.

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#67

Post by klabb3 »

avmich wrote: post #59
moneywoes wrote: post #55

Name rolls of the tongue

Yeah, when the Web was young, and people told URLs to each other, pronouncing "www", which was almost always the prefix of any web server host name also sounded funny.

Not sure what you’re talking about. Eitch tee tee pee ess colon slash slash doubleview doubleview doubleview dot just rolls off the tongue so easily.

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#68

Post by adam_arthur »

throwifasd wrote: post #63
adam_arthur wrote: post #61

Earlier quoted context omitted.

Though if true, it does beg the question, how can these services possibly be offered profitably if a single query requires many high end GPUs to run? Doesn't really track logically either. Though I'm not in the know. Perhaps ChatGPT is run with massive operating losses and its all VC subsidized.

This post [0] is a good primer and they are talking about a 20b model gpt3 has 175b. Their other posts go more into details but yes, it really is a massive operation. Some hard facts from here [1] talking about BLOOM with 175b parameters.. >> Installing the full 175B version is a challenge though as it requires around 350GB of GPU VRAM, which is not something one can easily afford. But hey why deploy a model like cha…

So it sounds like this is a question of loading the model into VRAM, and not a question of the cost of a single query. I assume once a model is loaded, many queries can be serviced by that model quickly.

There's nothing incorrect about my assertion. If it were to actually take many GPUs to service one query, then there is no mass scale cost viable consumer product. That's just a clear economic fact. Regardless if a model could be theoretically spun up in a cost inefficient manner.

And even 100s of GB of VRAM is not far off from consumer hardware. Look at how quickly graphics ram has expanded over time. About ~10x in ~10 years for high end cards, at a cursory glance at various Nvidia cards. At the same trajectory we could see a 400GB vram card within the next decade (though lots of assumptions)

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#69

Post by johnc1 »

adam_arthur wrote: post #38

So what's the level of effort to create ChatGPT equivalent products? Is it something where we'll have 100s of competing AIs, or is it gated to only a few large companies? Not up to date on current training/querying costs. Can these models feasibly be run locally? Given the large number of competitors already announced to ChatGPT, I fail to see how the space will be easily defensible or monetizable (despite large valu…

> Can these models feasibly be run locally?

Actually you can, it even works without GPU, here's a guide on running BLOOM (the open-source GPT-3 competitor of similar size) locally: https://towardsdatascience.com/run-bloom-the-largest-open-ac...

The problem is performance: - if you have GPUs with > 330GB VRAM, it'll run fast - otherwise, you'll run from RAM or NVMe, but very slowly - generating one token every few minutes or so (depending on RAM size / NVMe speed)

The future might be brighter: fp8 already exists and halves the RAM requirements (although it's still very hard to get it running), and there is ongoing research on fp4. Even that would still require 84GB of VRAM to run...

Re: ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model

#70

Post by corobo »

wokwokwok wrote: post #42
adam_arthur wrote: post #38

So what's the level of effort to create ChatGPT equivalent products? Is it something where we'll have 100s of competing AIs, or is it gated to only a few large companies? Not up to date on current training/querying costs. Can these models feasibly be run locally? Given the large number of competitors already announced to ChatGPT, I fail to see how the space will be easily defensible or monetizable (despite large valu…

> Can these models feasibly be run locally? Bluntly, no. The models which are small enough to run locally perform so badly it’s not worth bothering. To run inference on the large models the perform decently you need the equivalent of two or three top end graphics cards. If you're serious about looking into it now, consider looking at this project that lets you run a bunch of independent machines as a cluster for infe…

How badly is bad? What sort of output are we talking?

I am asking as I once had a Markov-chain IRC bot* and while it often struggled to string together a sentence, it was quite hilarious sometimes. Absolutely pointless other than the occasional laugh.

Can it form sentences or are those small models completely unusable for anything?

I'm not thinking OpenAI level uses - sort of compare a Postgres cluster to a SQLite file (not literally, conceptually I guess). Can it be used for single tasks in any way?

Could it figure out how to map search terms to URLs for a knowledge base type thing?

Forgive me if these are silly questions. The extent of my knowledge in this field is asking ChatGPT questions and going "that's so cool" when it answers.

* Your phone's predictive text except it finishes the sentence itself based on a word someone in chat used so that it felt on topic.

In my case it also learned how to form sentences from other people talking in chat, in hindsight it's amazing I never had a Tay issue.

https://en.m.wikipedia.org/wiki/Tay_(bot)

Post reply on HN