Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DenseFormer: Enhancing Information Flow in Transformers

arxiv.org

Post reply on HN
Print view View original

11–20 of 35 posts

Re: DenseFormer: Enhancing Information Flow in Transformers

#11

Post by tbalsam »

This is a very interesting idea, with DenseNets there are oftentimes some terrible memory gotchas that have gotten me over the past 7-8 years or so, so a part of me is sorta leaning back waiting for some memory usage shoe to drop not specified in the paper (even with the activation patterns!)

However, maybe this is not the case. I have a bit of a history of messing with residuals in neural networks, seeing more work on it is good. Fast training networks of course are a very slightly mild obsession of mine as well, and very useful to the field. Here's hoping it pans out as a motif, curious to see where it goes.

Re: DenseFormer: Enhancing Information Flow in Transformers

#12

Post by p1esk »

ml_basics wrote: post #5
p1esk wrote: post #3

This method has only been tested on tiny models (<1B) and tiny dataset (17B tokens). It’s not clear if it scales.

To be fair to the authors they are affiliated with a university and not a big industrial lab, so they may be working with significantly constrained resources. Not sure exactly what the best solution is for this case given that it affects most people outside of a very select few.

They could partner with big industrial labs.

Re: DenseFormer: Enhancing Information Flow in Transformers

#13

Post by valine »

The architecture changes are very straight forward. Model merging has shown that pre-trained transformer layers are very robust. I’ll bet it’s possible to fine tune a pre-trained model like mistral to use this architecture. That would enable someone to test it with more parameters without training a whole new base model.

Re: DenseFormer: Enhancing Information Flow in Transformers

#14

Post by Buttons840 »

p1esk wrote: post #3

This method has only been tested on tiny models (<1B) and tiny dataset (17B tokens). It’s not clear if it scales.

If a genie appeared and granted one wish, I would wish that we find an extremely powerful machine learning technique that doesn't scale. Imagine if an average desktop computer was almost as good as a billion dollar super computer.

In other words, I don't really care if it scales. I almost hope it doesn't.

Re: DenseFormer: Enhancing Information Flow in Transformers

#15

Post by quotemstr »

samus wrote: post #9
ml_basics wrote: post #4

Cool paper. Really interesting to see how even quite straightforward architectural modifications haven't yet all been exhausted yet, despite all the resources being poured into LLMs

The problem is that they have to be tested for 7B models at least to show promise for larger models. And that requires significant compute resources.

I wonder whether we're missing out on techniques that work well on large models but that don't show promise on small ones

Re: DenseFormer: Enhancing Information Flow in Transformers

#17

Post by p1esk »

Buttons840 wrote: post #14
p1esk wrote: post #3

This method has only been tested on tiny models (<1B) and tiny dataset (17B tokens). It’s not clear if it scales.

If a genie appeared and granted one wish, I would wish that we find an extremely powerful machine learning technique that doesn't scale. Imagine if an average desktop computer was almost as good as a billion dollar super computer. In other words, I don't really care if it scales. I almost hope it doesn't.

Not sure I understand what you mean by “doesn’t scale”. Are you trying to say you would like to see a tiny model performing as well as a large model?

Re: DenseFormer: Enhancing Information Flow in Transformers

#18

Post by hackerlight »

quotemstr wrote: post #15
samus wrote: post #9

Earlier quoted context omitted.

The problem is that they have to be tested for 7B models at least to show promise for larger models. And that requires significant compute resources.

I wonder whether we're missing out on techniques that work well on large models but that don't show promise on small ones

More like we're missing out on techniques full stop. Proving things at scale is GPU expensive and gatekeeps publication and therefore accessibility.

Re: DenseFormer: Enhancing Information Flow in Transformers

#19

Post by numeri »

valine wrote: post #13

The architecture changes are very straight forward. Model merging has shown that pre-trained transformer layers are very robust. I’ll bet it’s possible to fine tune a pre-trained model like mistral to use this architecture. That would enable someone to test it with more parameters without training a whole new base model.

They try this in the appendix without success, unfortunately. It seems having this enabled early on in training is important.
Post reply on HN