Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Weight-sparse transformers have interpretable circuits [pdf]

cdn.openai.com

Post reply on HN
Print view View original

41–48 of 48 posts

Re: Weight-sparse transformers have interpretable circuits [pdf]

#41

Post by astrange »

leogao wrote: post #25
lambdaone wrote: post #4

I find this fascinating, as it raises the possibility of a single framework that can unify neural and symbolic computation by "defuzzing" activations into what are effectively symbols. Has anyone looked at the possibility of going the other way, by fuzzifying logical computation?

There's been some work (e.g RASP - https://arxiv.org/abs/2106.06981 ) on taking logical computations and compiling them into transformer weights.

Sakana AI is also working on merging different transformer models together to combine skills.

https://sakana.ai/evolutionary-model-merge/

Re: Weight-sparse transformers have interpretable circuits [pdf]

#42

Post by ted_dunning »

p1esk wrote: post #35
m_ke wrote: post #31

Earlier quoted context omitted.

Before we had proper GPUs everyone said the same thing about Neural Networks. Current model architectures are optimized to get the most out of GPUs, which is why we have transformers dominating as they're mostly large dense matrix multiplies. There's plenty of work showing transformers improve with inner dimension size but it's not feasible to scale them up further because it blows up parameter and activation sizes (…

Yes, we know that large dense layers work better than small dense layers (up to a point). We also know how to train large dense models and then prune them. But we don’t know how to train large sparse models to be better than large dense models. If someone figures it out then we can talk about building hardware for it.

It isn't directly what you are asking for, but there is a similar relationship at work with respect to L_1 versus L_2 regularization. The number of samples required to train a model is O(log(d)) for L_1 and O(d) for L_2 where d is the dimensionality [1]. This relates to the standard random matrix results about how you can approximate high dimensional vectors in a log(d) space with (probably) small error.

At a very handwaving level, it seems reasonable that moving from L_1 to L_0 would have a similar relationship in learning complexity, but I don't think that has every been addressed formally.

[1] https://www.andrewng.org/publications/feature-selection-l1-v...

Re: Weight-sparse transformers have interpretable circuits [pdf]

#43

Post by yvdriess »

p1esk wrote: post #40
yvdriess wrote: post #39

Earlier quoted context omitted.

What do you mean by work better here? If it's for better accuracy then no they are not better at the same weight dimensions. The big thing is that sparse models allow you to train models with significantly larger dimensionality, blowing up the dimensions several orders of magnitudes. More dimensions leading to better results does not seem to be under a lot of contention, the open questions are more about quantifying…

The big thing is that sparse models allow you to train models with significantly larger dimensionality, blowing up the dimensions several orders of magnitudes. Do you have any evidence to support this statement? Or are you imagining some not yet invented algorithms running on some not yet invented hardware?

Sparse matrices can increase in dimension while keeping the same number of non-zeroes, that part is self evident. Sparse weights models can be trained, you probably are already aware of RigL and SRigL, there is similar other related work on unstructured and structured sparse training. You could argue that those adapt their algorithm to be executable on GPUs and that none are training at x100 or x1000 dimensions. Yes, that is the part that requires access to sparse compute hardware acceleration, which exists as prototypes [1] or are extremely expensive (Cerebras).

[1] https://dl.acm.org/doi/10.1109/MM.2023.3295848

Re: Weight-sparse transformers have interpretable circuits [pdf]

#44

Post by p1esk »

yvdriess wrote: post #43
p1esk wrote: post #40

Earlier quoted context omitted.

The big thing is that sparse models allow you to train models with significantly larger dimensionality, blowing up the dimensions several orders of magnitudes. Do you have any evidence to support this statement? Or are you imagining some not yet invented algorithms running on some not yet invented hardware?

Sparse matrices can increase in dimension while keeping the same number of non-zeroes, that part is self evident. Sparse weights models can be trained, you probably are already aware of RigL and SRigL, there is similar other related work on unstructured and structured sparse training. You could argue that those adapt their algorithm to be executable on GPUs and that none are training at x100 or x1000 dimensions. Yes,…

Unstructured sparsity cannot be implemented in hardware efficiently if you still want to do matrix multiplication. If you don’t want to do matrix multiplication you first need to come up with new algorithms, tested in software. This reminds me of what Numenta tried to do with their SDRs - note they didn’t quite succeed.

Re: Weight-sparse transformers have interpretable circuits [pdf]

#45

Post by yvdriess »

p1esk wrote: post #44
yvdriess wrote: post #43

Earlier quoted context omitted.

Sparse matrices can increase in dimension while keeping the same number of non-zeroes, that part is self evident. Sparse weights models can be trained, you probably are already aware of RigL and SRigL, there is similar other related work on unstructured and structured sparse training. You could argue that those adapt their algorithm to be executable on GPUs and that none are training at x100 or x1000 dimensions. Yes,…

Unstructured sparsity cannot be implemented in hardware efficiently if you still want to do matrix multiplication. If you don’t want to do matrix multiplication you first need to come up with new algorithms, tested in software. This reminds me of what Numenta tried to do with their SDRs - note they didn’t quite succeed.

> Unstructured sparsity cannot be implemented in hardware efficiently if you still want to do matrix multiplication.

Hard disagree. It certainly is a magnitude harder to design hardware for sp x sp MM, yes; it requires a paradigm shift to do sparse compute efficiently, but there are hardware architectures both in research and commercially available that do it efficiently. The same kind of architecture is needed to scale op graph compute. You see solutions at the smaller scale in FPGA and reconfigurable/dataflow accelerators, larger scale in Intel's PIUMA and Cerebras. I've been involved in co-design work of Graphblas on the software side and one of the aforementioned hardware platforms: the main issue with developing SpMSpM hardware lies more with the necessary capital and engineering investments being prioritized to current frontier AI model accelerators, not because of lack of proven results.

Re: Weight-sparse transformers have interpretable circuits [pdf]

#46

Post by yvdriess »

leogao wrote: post #30
esafak wrote: post #23

Earlier quoted context omitted.

As the lead author, why do you think so?

I'm not an expert at hardware, so take this with a grain of salt, but there are two main reasons: - Discrete optimisation is always going to be harder than continuous optimization. Learning the right sparsity mask is fundamentally a very discrete operation. So even just matching fully continuous dense models in optimization efficiency is likely to be difficult. Though perhaps we can get some hope from the fact that M…

A note on the hardware part: it does not require NN-specific hardware akin to neuromorphic. Sparse compute oriented architectures already have been developer for other reasons, such as large scale graph analysis or inference. It will still require significant effort to use it to train large models, but it would not be starting from scratch.

Re: Weight-sparse transformers have interpretable circuits [pdf]

#47

Post by yvdriess »

kwillets wrote: post #36
m_ke wrote: post #9

We really need new hardware optimized for sparse compute. Deep Learning models would work way better with much higher dimensional sparse vectors but current hardware only excels at dense GMMs and structured sparsity.

My last dive into matrix computations was years ago, but the need was the same back then. We could sparsify matrices pretty easily, but the infrastructure was lacking. Some things never change.

On the software side I can recommend https://github.com/DrTimothyAldenDavis/GraphBLAS It is hard to make a sparse linear algebra framework, but Tim Davis has been doing a great job collecting the various optimal algorithms I to a single framework that acts more like an algebra than a collection of kernels.

Re: Weight-sparse transformers have interpretable circuits [pdf]

#48

Post by pedrosorio »

lambdaone wrote: post #4

I find this fascinating, as it raises the possibility of a single framework that can unify neural and symbolic computation by "defuzzing" activations into what are effectively symbols. Has anyone looked at the possibility of going the other way, by fuzzifying logical computation?

That was a bust: https://www.reddit.com/r/engineering/comments/pwht4f/whateve...
Post reply on HN