DenseFormer: Enhancing Information Flow in Transformers
1–10 of 35 posts
Re: DenseFormer: Enhancing Information Flow in Transformers
#2Even better is the result on page 7 that perplexity drops faster by wall-clock time. Even if you're getting fewer iterations per hour of rented GPU time, you're still coming out ahead in model performance.
Re: DenseFormer: Enhancing Information Flow in Transformers
#3This method has only been tested on tiny models (<1B) and tiny dataset (17B tokens). It’s not clear if it scales.
Re: DenseFormer: Enhancing Information Flow in Transformers
#4Cool paper. Really interesting to see how even quite straightforward architectural modifications haven't yet all been exhausted yet, despite all the resources being poured into LLMs
Re: DenseFormer: Enhancing Information Flow in Transformers
#5This method has only been tested on tiny models (<1B) and tiny dataset (17B tokens). It’s not clear if it scales.
To be fair to the authors they are affiliated with a university and not a big industrial lab, so they may be working with significantly constrained resources. Not sure exactly what the best solution is for this case given that it affects most people outside of a very select few.
Re: DenseFormer: Enhancing Information Flow in Transformers
#6This method has only been tested on tiny models (<1B) and tiny dataset (17B tokens). It’s not clear if it scales.
But it may scale -- that's science in progress
Re: DenseFormer: Enhancing Information Flow in Transformers
#7> Impact statement:
> This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
I found this particularly charming.
Re: DenseFormer: Enhancing Information Flow in Transformers
#8> Impact statement: > This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here. I found this particularly charming.
AFAIK this was the default, copy paste impact statement by ICML template.
Re: DenseFormer: Enhancing Information Flow in Transformers
#9Cool paper. Really interesting to see how even quite straightforward architectural modifications haven't yet all been exhausted yet, despite all the resources being poured into LLMs
The problem is that they have to be tested for 7B models at least to show promise for larger models. And that requires significant compute resources.
Re: DenseFormer: Enhancing Information Flow in Transformers
#10Cool paper. Really interesting to see how even quite straightforward architectural modifications haven't yet all been exhausted yet, despite all the resources being poured into LLMs
The problem is that they have to be tested for 7B models at least to show promise for larger models. And that requires significant compute resources.
Due to some of my personal experiences over the years w/ model development, I believe that this is more due to a failure of the current mainline version of Transformers (the ++ version I believe) not scaling properly, vs an indicator of scale.
If that is the case, then it may well be possible to fix some of the scaling issues more apparent with smaller transformer models (maybe not, though). This is at least some of the reasoning that I've been applying when developing hlb-gpt, for example. It's partially also why I think changing how we use nonlinearities within the network might impact scaling, due to some of the activation spikes used in more linear regions of the network to control network behavior in a way not originally intended.
Agreed that it does require a ton of resources though. But I do think that the problem can be solved on a smaller scale. If we don't have a cleanly logarithmic curve, then I think that something is dearly wrong with our base architecture. (However, of course, I may entirely be missing something here).