Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RWKV: Reinventing RNNs for the Transformer Era

arxiv.org

Post reply on HN
Print view View original

31–40 of 176 posts

Re: RWKV: Reinventing RNNs for the Transformer Era

#31

Post by pico_creator »

pico_creator wrote: post #29
bigdict wrote: post #7

Jesus H. Christ, first time I see a collaboration this big on a ML paper. How does a team like that even come together? This isn't the LHC.

RWKV has been running in public (relatively obscure to other models) for the past 2 years Mostly lead by a single person (blink). This community consist mostly of people outside the academia / big VC tech scene When eleutherAI offered to help us with writing the paper. Various key folks banded together for the paper, as it’s what seems to be a very strong alternative to transformers This does not mean everyone in the…

AKA: no one here is incentivised to fight for bigger ownership, no promotion or KPI or pay was on the line

Re: RWKV: Reinventing RNNs for the Transformer Era

#32

Post by ftxbro »

pico_creator wrote: post #23

Hi Everyone, I'm a regular involved with the RWKV community. AMA, on RWKV, and I will do my best to answer them here for the next hour (one at a time) PS: you can find our discord here : https://discord.gg/qt9egFA7ve

The paper says it's comparable to transformers right now but that means that it might be better later. Do you guys have concrete plans to make it better? Are they secret? Also, what's the deal with that foundation? Is it a cult or like the new OpenAI that will turn closed or maybe it's to reap the value of random contributors to the project?

Re: RWKV: Reinventing RNNs for the Transformer Era

#33

Post by pico_creator »

whimsicalism wrote: post #2

Awesome to see this published. Work on transformer alternatives, especially parallelizable ones like this, is incredibly important - it would suck if we get sucked down a local optima in architecture without actually looking at nearby viable alternatives.

Yup. I’m all here for infinite scaling of context size

Re: RWKV: Reinventing RNNs for the Transformer Era

#34

Post by rahimnathwani »

Buttons840 wrote: post #19
ftxbro wrote: post #9

Earlier quoted context omitted.

I think it's just, everyone in their discord channel.

What channel? I have an application for sequence models I think might be novel, and I'd like to be able to get credit and help research it if possible. Probably somebody has already done it, but I cannot search well enough to find related literature.

> What channel?

https://discord.gg/qt9egFA7ve

Re: RWKV: Reinventing RNNs for the Transformer Era

#35

Post by pico_creator »

cs702 wrote: post #22

Alas, it doesn't appear to work well for longer contexts: https://twitter.com/arankomatsuzaki/status/16390003799784038... Has anyone here experimented with this recently to confirm?

It has already been confirmed that with the right dataset we can scale it effectively from 2k to 4K, and 4K to 8k via fine tuning (you dun even need to train a new foundation model)

We believe this can be done for 16k to way beyond 100k

Research in how RWKV handle the hidden state shows that it is barely used (imo: (This is actively being experimented on - we dun really know the limit yet)

Re: RWKV: Reinventing RNNs for the Transformer Era

#36

Post by rsfern »

bee_rider wrote: post #5

Dumb arxiv question (sorry); is it possible to see what journal a paper was actually submitted to, to help find the reviewed version when it comes out?

Sometimes that is included in the journal LaTeX template, but otherwise authors (in my field at least) typically won’t say where a paper is submitted before it’s accepted for publication there, or at the very least if it’s made it past the editorial rejection hurdle and been sent out for review

Probably your best bet for getting a feed heads up is to set a GScholar alert on the first author?

Re: RWKV: Reinventing RNNs for the Transformer Era

#37

Post by FallDead »

ftxbro wrote: post #32
pico_creator wrote: post #23

Hi Everyone, I'm a regular involved with the RWKV community. AMA, on RWKV, and I will do my best to answer them here for the next hour (one at a time) PS: you can find our discord here : https://discord.gg/qt9egFA7ve

The paper says it's comparable to transformers right now but that means that it might be better later. Do you guys have concrete plans to make it better? Are they secret? Also, what's the deal with that foundation? Is it a cult or like the new OpenAI that will turn closed or maybe it's to reap the value of random contributors to the project?

(Note: My comments do not represent or project those of my collaborators) I remember talking to Blink DL about this, I think the plan is just to build an ecosystem, provide more diversity in the DL space. There are plans to make a RWKV5, they are in the open in the RWKV5 channel. From an engineering standpoint I don't really see the "reap" the value of random contributors to the project. Most of us I believe ... are hackers and tinkerers that just want to learn and contribute and be apart of something that can change the current

Re: RWKV: Reinventing RNNs for the Transformer Era

#38

Post by pico_creator »

ftxbro wrote: post #32
pico_creator wrote: post #23

Hi Everyone, I'm a regular involved with the RWKV community. AMA, on RWKV, and I will do my best to answer them here for the next hour (one at a time) PS: you can find our discord here : https://discord.gg/qt9egFA7ve

The paper says it's comparable to transformers right now but that means that it might be better later. Do you guys have concrete plans to make it better? Are they secret? Also, what's the deal with that foundation? Is it a cult or like the new OpenAI that will turn closed or maybe it's to reap the value of random contributors to the project?

Completely the opposite.

- it is NOT backed directly or owned by any VC funded company

- it is 100% OSS driven by the community (Apache 2 license)

- it’s currently the top OSS chat model that can be used commercially on the chatbot arena score board

- IMO it is undertrained, so expanding the training data alone will make it much better (however for the sake of this paper, we wanted to focus on architecture not training data, so we compared similarly trained models)

And yes we do have multiple experiments and plans to make it better. It’s a list, and we will not know which is final until we try. Individual members can go to great lengths on what they are working on

For better or worse, being truly OSS means our initiatives are more disorganized then a centrally planned org

Re: RWKV: Reinventing RNNs for the Transformer Era

#39

Post by stellaathena »

bee_rider wrote: post #5

Dumb arxiv question (sorry); is it possible to see what journal a paper was actually submitted to, to help find the reviewed version when it comes out?

The paper is going to be submitted to EMNLP next month. An early version is being released now to garner feedback and improve the paper before submission.

Re: RWKV: Reinventing RNNs for the Transformer Era

#40

Post by evolve7942 »

bigdict wrote: post #7

Jesus H. Christ, first time I see a collaboration this big on a ML paper. How does a team like that even come together? This isn't the LHC.

If you think this is wild, see the PaLM 2 paper with 2.5 pages of 2 column attributions.

https://arxiv.org/pdf/2305.10403.pdf

Post reply on HN