Earlier quoted context omitted.
They could partner with big industrial labs.
Well now that they have a promising result, maybe.
They had this promising result before they posted the paper.
DenseFormer: Enhancing Information Flow in Transformers
31–35 of 35 posts
Re: DenseFormer: Enhancing Information Flow in Transformers
#32Earlier quoted context omitted.
They could partner with big industrial labs.
Nah, nobody's begging for people to A) come use time on their GPUs B) come watch them train their biggest models. Nor does it make sense to spend $X00M training a big model using an experimental technique before you announce it, nor does it make sense to hold back breakthroughs as an academic until someone commercializes it at scale. Category error.
I do ML research at a small industrial lab. I’ll gladly provide some compute to people with a cool idea if that results in my company name listed on a paper in a top conference. Especially if the people are from a top university.
Re: DenseFormer: Enhancing Information Flow in Transformers
#33Earlier quoted context omitted.
If a genie appeared and granted one wish, I would wish that we find an extremely powerful machine learning technique that doesn't scale. Imagine if an average desktop computer was almost as good as a billion dollar super computer. In other words, I don't really care if it scales. I almost hope it doesn't.
Even pocket computers (smartphones) are already better than billion dollar supercomputers from decades past. What is your point?
That no one has an advantage
Re: DenseFormer: Enhancing Information Flow in Transformers
#34I'm one of the authors, happy to answer questions.
Is it possible to combine your approach with NATTEN? It seems that both approaches are optimizing from different directions and can be combined with significant throughput and small performance improvements?
Re: DenseFormer: Enhancing Information Flow in Transformers
#35Earlier quoted context omitted.
They try this in the appendix without success, unfortunately. It seems having this enabled early on in training is important.
We're still working on training the DWA weights on top of a pretained model. We're hopeful that this is feasible. The experiments you're mentioning in the appendix are not changing the learning rate scheduler. E.g., when starting to train the DWA weights after 20k iterations, the learning rate is already quite small. To some extent, this might explain the diminishing returns. Maybe this could work with a completely d…
Yeah, you can't change the model much with low LRs. That's the point! Same reason you don't get continual-learning if you just keep using low LRs: https://arxiv.org/abs/2403.08763 You need to really shake up the model if you want to learn some genuinely better (ie. different) internal representations that exploits the DenseNet (https://arxiv.org/abs/1608.06993)/LTG-BERT (https://arxiv.org/abs/2311.02265) arch you're using here.