Viewing profile — pico_creator

(original article author) I view it more as a shortcut. We have trained 7B and 14B models from scratch, matching transformer performance with similar sized datasets. This has been …

There is work done for Vision RWKV, and audio RWKV, an example paper is here: https://arxiv.org/abs/2403.02308 Its the same principle as open transformer models where an adapter is…

One of the interesting "new direction" for RWKV and Mamba (or any recurrent model), is the monitoring and manipulation of the state in between token. For steerability, alignment, e…

Not sure how indepth you want it to be. But we did do a co-presentation with one of the coauthors of mamba at latent space : https://www.youtube.com/watch?v=LPe6iC73lrc

There is a current lack of "O1 style" reasoning dataset in open source space. QWQ did not release their dataset. So that would take some time for the community to prepare. It's def…

kinda on a todo list, the model is open source on HF for anyone who is willing to make it work with lmarena

lower compute cost especially over longer sequence length. Depending on context length, its 10x, 100x, or even 1000x+ cheaper. (quadratic vs linear cost difference)

RWKV already solve the parallel compute problem for GPU, based on the changes it has done - so it is a recurrent model that can scale to thousands++ of GPU no issue. Arguably with …

Currently the strongest RWKV model is 32B in size: https://substack.recursal.ai/p/q-rwkv-6-32b-instruct-preview This is a full drop in replacement for any transformer model use cas…

Hey there, im Eugene / PicoCreator - co-leading the RWKV project - feel free to AMA =)

This is actually the hypothesis for cartesia (state space team), and hence their deep focus on voice model specifically. Taking full advantage of recurrent models constant time com…

Not an MoE, but we have already done hybrid models. And found it to be highly performant (as per the training budget) https://arxiv.org/abs/2407.12077

Someone is losing the money. It’s elaborated in the article how and why this happens TLDR, VC money, is being burnt/lost

Im quite sure there is more than a 100 clusters even. Though that would be harder to prove. So yea, it would be rough

I actually signed up for separate new account, to double check that my business account was not being favored or rigged in "private beta" Its really not that hard to validate this …

Not at $0.5 (which the lower bound in their marketing), but $1.5 is very doable on right times (done so multiple times) The article says $2. Which is quite consistent for a small c…

Yup, but they at-least know where all these "small unused clusters" are. Bag holders, do not want to be shouting to the world they are bag holders.

Also: how many of those consultants, have actually rented GPU's - used them for inference - or used them to finetune / train

Do we have actual fp8 numbers? (or i could proxy it by /2 the fp4)

Feel free to forward to the clients of "paid consultant". Also how do i collect my cut.

Given their rising stock price trend, due to their moves in AI. Definitely worth it for them

I really suggest shopping around. <$2 SXM is a real thing, if your patient enough on the schedule.

Makes sense, though only folks like runpod / sfcompute / etc, have enough visibility to maybe pull this off? Its a risker move - then just taxing the excess compute now, and print …

Only if ur a collector (so no if ur plugging it in)