Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Eagle 7B: Soaring past Transformers

blog.rwkv.com

Post reply on HN
Print view View original

81–86 of 86 posts

Re: Eagle 7B: Soaring past Transformers

#81

Post by shawn-butler »

klipt wrote: post #37
mediaman wrote: post #36

Earlier quoted context omitted.

Knowledge transfer over generations is a function of the brain. Other species have much more limited ability to transfer knowledge intergenerationally, and that is because the human brain's capability for symbolic language is much more advanced than other animals', who are not able to encode knowledge nearly as efficiently.

The point is it's a function of many connected brains, not just one brain.

What if it's all the same electron[0]?

[0]: https://www.nobelprize.org/prizes/physics/1965/feynman/lectu...

Re: Eagle 7B: Soaring past Transformers

#82

Post by IanCal »

visarga wrote: post #8

This shows the model architecture, be it transformer, Mamba, SSM or RWKV - doesn't really matter when compared to the impact of the training set. We're spending too much time debating models when we should be talking about language data, a reservoir of human experience won at great sacrifice by humanity. And the same data when used to train humans creates modern capable people. Alone, without society and language, we…

I expect this is why openai were focusing on partnering with people with high quality data

https://openai.com/blog/data-partnerships

Re: Eagle 7B: Soaring past Transformers

#83

Post by [deleted] »

wokwokwok wrote: post #27
Szpadel wrote: post #15

Earlier quoted context omitted.

RWKV does not have context size, or in other way do look at it, it does have infinite one. As far as I understand this, there is internal state that holds new information while reading input, later information can overwrite previous ones with is arguably human like behaviour.

One of three things has to be true. Either: a) this is false b) perfect recall is false (ie. as the internal state is overwritten, you lose information about previous entries in the context) c) the inference time scales by the context length. It’s not possible to have perfect recall over an arbitrary length in fixed time. Not hard. Totally not possible at all That would mean you can scan an infinite amount of data pe…

[deleted]

Re: Eagle 7B: Soaring past Transformers

#84

Post by jeremycochoy »

karterk wrote: post #5

It's interesting how all focus is now primarily on decoder-only next-token-prediction models. Encoders (BERT, encoder of T5) are still useful for generating embedding for tasks like retrieval or classification. While there is a lot of work on fine-tuning BERT and T5 for such tasks, it would be nice to see more research on better pre-training architectures for embedding use cases.

I believe RWKV is actually an architecture that can be used for encoding: given a LSTM/GRU, you can simply take the last state as an encoding of your sequence. The same should be possible with RWKV, right?

Re: Eagle 7B: Soaring past Transformers

#86

Post by viktor-ferenczi »

After reading all these, I've landed at the following conclusion:

Regardless of its architecture there is only a finite amount of information the language model can work with at any given time. It depends on the task at hand which way of "forgetting" causes the least problems.

For coding and math a perfect context with a well defined maximum length of 16k..256k tokens paired with high quality ICL would work better than automated "random" forgetting. However, it requires a good strategy to present only the information relevant for the task to fit into the maximum context length.

For free-form literature and other non-technical stuff automated forgetting is likely beneficial, because you don't need to come up with a strategy to choose what's important to keep in-context. What you get is automated gradual forgetting and "mixing up" past memories, just like in humans.

Since I'm a software developer geek I strongly prefer the first one, but as you can see, it depends on the task at hand.

Post reply on HN