41–50 of 176 posts
Jesus H. Christ, first time I see a collaboration this big on a ML paper. How does a team like that even come together? This isn't the LHC.
I think it's just, everyone in their discord channel.
Earlier quoted context omitted.
What channel? I have an application for sequence models I think might be novel, and I'd like to be able to get credit and help research it if possible. Probably somebody has already done it, but I cannot search well enough to find related literature.
Write it and submit it yourself. Sole authorship is given more weight these days for whatever reason. In a multi author publication, even if you are first author or listed as equal contribution, if there is a more famous person on the paper everyone will assume they did it.
One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…
Hopefully this gets the point across without the mathematics and a precise description.
Earlier quoted context omitted.
The paper says it's comparable to transformers right now but that means that it might be better later. Do you guys have concrete plans to make it better? Are they secret? Also, what's the deal with that foundation? Is it a cult or like the new OpenAI that will turn closed or maybe it's to reap the value of random contributors to the project?
Completely the opposite. - it is NOT backed directly or owned by any VC funded company - it is 100% OSS driven by the community (Apache 2 license) - it’s currently the top OSS chat model that can be used commercially on the chatbot arena score board - IMO it is undertrained, so expanding the training data alone will make it much better (however for the sake of this paper, we wanted to focus on architecture not traini…
Hi Everyone, I'm a regular involved with the RWKV community. AMA, on RWKV, and I will do my best to answer them here for the next hour (one at a time) PS: you can find our discord here : https://discord.gg/qt9egFA7ve
One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer quest…
And this apply to all models. So when a specific model now does badly even at 32k or 50k it’s hard to say if it’s an architecture design issue, or a dataset issue
Dumb arxiv question (sorry); is it possible to see what journal a paper was actually submitted to, to help find the reviewed version when it comes out?
Earlier quoted context omitted.
Write it and submit it yourself. Sole authorship is given more weight these days for whatever reason. In a multi author publication, even if you are first author or listed as equal contribution, if there is a more famous person on the paper everyone will assume they did it.
Write it and submit it to who? I'm not familiar enough with the field to find any prior work or related work.
Earlier quoted context omitted.
Completely the opposite. - it is NOT backed directly or owned by any VC funded company - it is 100% OSS driven by the community (Apache 2 license) - it’s currently the top OSS chat model that can be used commercially on the chatbot arena score board - IMO it is undertrained, so expanding the training data alone will make it much better (however for the sake of this paper, we wanted to focus on architecture not traini…
Can beginners without money with by PhDs, contribute? If so what would be the best way to start?