Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kolmogorov-Arnold Networks

github.com

Post reply on HN
Print view View original

61–70 of 149 posts

Re: Kolmogorov-Arnold Networks

#61

Post by mindcrime »

krasin wrote: post #15
DeathArrow wrote: post #9

Earlier quoted context omitted.

>some algorithmic advance will enhance LLMs by orders of magnitude I would worry if I'd own Nvidia shares.

Actually, that would be fantastic for NVIDIA shares; 1. A new architecture would make all/most of these upcoming Transformer accelerators obsolete => back to GPUs. 2. Higher performance LLMs on GPUs => we can speed up LLMs with 1T+ parameters. So, LLMs become more useful, so more of GPUs would be purchased.

1. A new architecture would make all/most of these upcoming Transformer accelerators obsolete => back to GPUs.

There's no guarantee that that is what would happen. The right (or wrong, depending on your POV) algorithmic breakthrough might make GPU's obsolete for AI, by making CPU's (or analog computing units, or DSP's, or "other") the preferred platform to run AI.

Re: Kolmogorov-Arnold Networks

#62

Post by Lichtso »

1. Interestingly the foundations of this approach and MLP were invented / discovered around the same time about 66 years ago:

1957: https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Arnold_repr...

1958: https://en.wikipedia.org/wiki/Multilayer_perceptron

2. Another advantage of this approach is that it has only one class of parameters (the coefficients of the local activation functions) as opposed to MLP which has three classes of parameters (weights, biases, and the globally uniform activation function).

3. Everybody is talking transformers. I want to see diffusion models with this approach.

Re: Kolmogorov-Arnold Networks

#63

Post by goggy_googy »

SpaceManNabs wrote: post #45

How does back propagation work now? Do these suffer from vanishing or exploding gradients?

No, the activations are a combination of the basis function and the spline function. It's a little unclear to me still how the grid works, but it seems like this shouldn't suffer anymore than a generic relu MLP.

Re: Kolmogorov-Arnold Networks

#64

Post by the8472 »

snewman wrote: post #60
ALittleLight wrote: post #7

I can't assess this, but I do worry that overnight some algorithmic advance will enhance LLMs by orders of magnitude and the next big model to get trained is suddenly 10,000x better than GPT-4 and nobody's ready for it.

I think this is unlikely. There has never (in the visible fossil record) been a mutation that suddenly made tigers an order of magnitude stronger and faster, or humans an order of magnitude more intelligent. It's been a long time (if ever?) since chip transistor density made a multiple-order-of-magnitude leap. Any complex optimized system has many limiting factors and it's unlikely that all of them would leap forward…

Comparing it to evolution and SNPs isn't really a good analogy. Novel network architectures are much larger changes, maybe comparable to new organelles or metabolic pathways? And those have caused catastrophic changes. Evolution also operates on much longer time-scales due to its blind parallel search.

https://en.wikipedia.org/wiki/Oxygen_catastrophe

Re: Kolmogorov-Arnold Networks

#65

Post by GistNoesis »

itsthecourier wrote: post #44
GistNoesis wrote: post #34

I quickly skimmed the paper, got inspired to simplify it, and created some Pytorch Layer : https://github.com/GistNoesis/FourierKAN/ The core is really just a few lines. In the paper they use some spline interpolation to represent 1d function that they sum. Their code seemed aimed at smaller sizes. Instead I chose a different representation, aka fourier coefficients that are used to interpolate the functions of indiv…

you really are a pragmatic programmer, Noesis

Thanks. I like simple things.

Sums and products can get you surprisingly far.

Conceptually it's simpler to think about and optimize. But you can also write it use einsum to do the sum product reductions (I've updated some comment to show how) to use less memory, but it's more intimidating.

You can probably use KeOps library to fuse it further (einsum would get in the way).

But the best is probably a custom kernel. Once you have written it as sums and product, it's just iterating. Like the core is 5 lines, but you have to add roughly 500 lines of low-level wrapping code to do cuda parallelisation, c++ to python, various types, manual derivatives. And then you have to add various checks so that there are no buffer overflows. And then you can optimize for special hardware operations like tensor cores. Making sure along the way that no numerical errors where introduced.

So there are a lot more efforts involved, and it's usually only worth it if the layer is promising, but hopefully AI should be able to autocomplete these soon.

Re: Kolmogorov-Arnold Networks

#67

Post by trwm »

Lichtso wrote: post #62

1. Interestingly the foundations of this approach and MLP were invented / discovered around the same time about 66 years ago: 1957: https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Arnold_repr... 1958: https://en.wikipedia.org/wiki/Multilayer_perceptron 2. Another advantage of this approach is that it has only one class of parameters (the coefficients of the local activation functions) as opposed to MLP which has thr…

Biases are just weights on an always on input.

There isn't much difference between weights of a linear sum and coefficients of a spline.

Re: Kolmogorov-Arnold Networks

#68

Post by brrrrrm »

doesn't KA representation require continuous univariate functions? do B-splines actually cover the space of all continuous functions? wouldn't... MLPs be better for the learnable activation functions?

Re: Kolmogorov-Arnold Networks

#69

Post by trwm »

wongarsu wrote: post #27
mxwsn wrote: post #11

From the preprint - 100 input dimensions is considered "high", and most problems considered have 5 or fewer input dimensions. This is typical of physics-inspired settings I've seen considered in ML. The next step would be demonstrating them on MNIST, which, at 784 dimensions is tiny by modern standards.

In actual business processes there are lots of ML problems with fewer than 100 input dimensions. But for most of them decision trees are still competitive with neural networks or even outperform them.

Business processes don't need deep learning in the first place. It is just there because hype.
Post reply on HN