Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DenseFormer: Enhancing Information Flow in Transformers

arxiv.org

Post reply on HN
Print view View original

21–30 of 35 posts

Re: DenseFormer: Enhancing Information Flow in Transformers

#21

Post by bilsbie »

valine wrote: post #13

The architecture changes are very straight forward. Model merging has shown that pre-trained transformer layers are very robust. I’ll bet it’s possible to fine tune a pre-trained model like mistral to use this architecture. That would enable someone to test it with more parameters without training a whole new base model.

I haven’t been able to make sense of model merging. Any insights?

Wouldn’t weights between models be completely different? And then there are architecture differences on top of that.

Re: DenseFormer: Enhancing Information Flow in Transformers

#22

Post by MacsHeadroom »

Buttons840 wrote: post #14
p1esk wrote: post #3

This method has only been tested on tiny models (<1B) and tiny dataset (17B tokens). It’s not clear if it scales.

If a genie appeared and granted one wish, I would wish that we find an extremely powerful machine learning technique that doesn't scale. Imagine if an average desktop computer was almost as good as a billion dollar super computer. In other words, I don't really care if it scales. I almost hope it doesn't.

Even pocket computers (smartphones) are already better than billion dollar supercomputers from decades past.

What is your point?

Re: DenseFormer: Enhancing Information Flow in Transformers

#23

Post by valine »

bilsbie wrote: post #21
valine wrote: post #13

The architecture changes are very straight forward. Model merging has shown that pre-trained transformer layers are very robust. I’ll bet it’s possible to fine tune a pre-trained model like mistral to use this architecture. That would enable someone to test it with more parameters without training a whole new base model.

I haven’t been able to make sense of model merging. Any insights? Wouldn’t weights between models be completely different? And then there are architecture differences on top of that.

Model merging is usually done with different fine-tunes of the same model. It doesn’t work if the base models are different.

One of the more surprising things is that you can actually repeat layers to improve model performance, ie 1-1-2-2 instead of 1-2. That’s how you get models with higher parameter counts than the original.

Re: DenseFormer: Enhancing Information Flow in Transformers

#24

Post by namibj »

valine wrote: post #23
bilsbie wrote: post #21

Earlier quoted context omitted.

I haven’t been able to make sense of model merging. Any insights? Wouldn’t weights between models be completely different? And then there are architecture differences on top of that.

Model merging is usually done with different fine-tunes of the same model. It doesn’t work if the base models are different. One of the more surprising things is that you can actually repeat layers to improve model performance, ie 1-1-2-2 instead of 1-2. That’s how you get models with higher parameter counts than the original.

C.f. also Universal Transformer: the same layer stacked a lot. The sparse version of that is basically MoE with also a stick-breaking mechanism to prevent vanishing gradient while letting the model decide whether to terminate layer-count at a token early (ofc with training rewards to favor less layers, to represent the compute savings).

Re: DenseFormer: Enhancing Information Flow in Transformers

#25

Post by refulgentis »

p1esk wrote: post #12
ml_basics wrote: post #5

Earlier quoted context omitted.

To be fair to the authors they are affiliated with a university and not a big industrial lab, so they may be working with significantly constrained resources. Not sure exactly what the best solution is for this case given that it affects most people outside of a very select few.

They could partner with big industrial labs.

Nah, nobody's begging for people to A) come use time on their GPUs B) come watch them train their biggest models. Nor does it make sense to spend $X00M training a big model using an experimental technique before you announce it, nor does it make sense to hold back breakthroughs as an academic until someone commercializes it at scale. Category error.

Re: DenseFormer: Enhancing Information Flow in Transformers

#26

Post by zwaps »

1. They compare with an older sort of standard implementation of a transformer Unsure whether the results would be equally significant compared to models with gated units or multiquery etc.

2. The difference seems to diminish with scale. Real life transformers obviously are much larger and train on many more tokens.

3. A very significant part of training transformer models are the throughoutput and memory optimizations. I wonder how their model would work with such fused kernels or specialized paged KV cache schemes. Or activation checkpointing, if run locally.

4. Indeed they claim no memory impact, but their code shows that their experiments are conducted with a special optimized version which requires all activations to reside in a single tensor at all times. Not sure this would work with 3d parallelism on multiple nodes etc.

Re: DenseFormer: Enhancing Information Flow in Transformers

#28

Post by sp332 »

p1esk wrote: post #12
ml_basics wrote: post #5

Earlier quoted context omitted.

To be fair to the authors they are affiliated with a university and not a big industrial lab, so they may be working with significantly constrained resources. Not sure exactly what the best solution is for this case given that it affects most people outside of a very select few.

They could partner with big industrial labs.

Well now that they have a promising result, maybe.

Re: DenseFormer: Enhancing Information Flow in Transformers

#30

Post by matteopagli »

numeri wrote: post #19
valine wrote: post #13

The architecture changes are very straight forward. Model merging has shown that pre-trained transformer layers are very robust. I’ll bet it’s possible to fine tune a pre-trained model like mistral to use this architecture. That would enable someone to test it with more parameters without training a whole new base model.

They try this in the appendix without success, unfortunately. It seems having this enabled early on in training is important.

We're still working on training the DWA weights on top of a pretained model. We're hopeful that this is feasible. The experiments you're mentioning in the appendix are not changing the learning rate scheduler. E.g., when starting to train the DWA weights after 20k iterations, the learning rate is already quite small. To some extent, this might explain the diminishing returns. Maybe this could work with a completely different learning rate scheduler.
Post reply on HN