Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kolmogorov-Arnold Networks

github.com

Post reply on HN
Print view View original

131–140 of 149 posts

Re: Kolmogorov-Arnold Networks

#131

Post by hansvm »

brrrrrm wrote: post #68

doesn't KA representation require continuous univariate functions? do B-splines actually cover the space of all continuous functions? wouldn't... MLPs be better for the learnable activation functions?

The paper [0] is pretty good for handling questions like those.

> doesn't KA representation require continuous univariate functions?

All multivariate continuous functions (on a bounded domain) can be represented as compositions of addition and univariate continuous functions. Much like an MLP, you can also approximate discontinuous functions well on most of the domain (learning a nearby continuous function instead).

> do B-splines actually cover the space of all continuous functions

Much like an MLP, you can hit your favorite accuracy bound with more control points.

> wouldn't... MLPs be better for the learnable activation functions

Perhaps. A B-spline is comparatively very fast to compute. Also, local training examples have global impacts on an MLP's weights. That's good and bad. One property you would expect while training a KAN in limited data regimes is that some control points are never updated, leading to poor generalization due to something like a phase shift as you cross over control points (I think the entropy-based regularizer they have in the paper probably solves that, but YMMV). The positive side of that coin is that you neatly side-step catastrophic forgetting.

[0] https://arxiv.org/abs/2404.19756

Re: Kolmogorov-Arnold Networks

#132

Post by droidlyx »

GistNoesis wrote: post #34

I quickly skimmed the paper, got inspired to simplify it, and created some Pytorch Layer : https://github.com/GistNoesis/FourierKAN/ The core is really just a few lines. In the paper they use some spline interpolation to represent 1d function that they sum. Their code seemed aimed at smaller sizes. Instead I chose a different representation, aka fourier coefficients that are used to interpolate the functions of indiv…

Hi Noesis, I just noticed that your implementation, combined with the efficientKAN by Blealtan (https://github.com/Blealtan/efficient-kan), results in a structure very similar to Siren(MLP with Sin activations). efficientKAN first computes the common basis functions for all the edge activations and the output can be calculated with a linear combination of the basis. If the basis functions are fourier, then a KAN layer can be viewed as a linear layer with fixed weights + Sin activation + a linear layer with learnable weights, which is a special form of Siren. I think this may show some connection between KAN and MLP.

Re: Kolmogorov-Arnold Networks

#135

Post by adityang5 »

Very cool stuff! Exciting to see so many people sharing their works on KANs. Seeing as the authors claim that KANs are able to reduce the issues of catastrophic forgetting that we see in MLPs, I thought "Wouldn't it be nice if there was an LLM that substituted MLPs with KANs?". I looked around and didn't find one, so I built one!

- PyTorch Module of the KAN GPT

- Deployed to PyPi

- MIT Licence

- Test Cases to ensure forward-backward passes work as expected

- Training script

I am currently working on training it on the WebText dataset to compare it to the original gpt2. Facing a few out-of-memory issues at the moment. Perhaps the vocab size (50257) is too large?

I'm open to contributions and would love to hear your thoughts!

https://github.com/AdityaNG/kan-gpt

https://pypi.org/project/kan-gpt/

Re: Kolmogorov-Arnold Networks

#136

Post by nu91 »

esafak wrote: post #123
nu91 wrote: post #99

I am curious to know if this type of network can help with causal inference.

They help with interpretability, which is a step brother. See for example "From Shapley Values to Generalized Additive Models and back" https://arxiv.org/abs/2209.04012

Indeed, Thank you!

Re: Kolmogorov-Arnold Networks

#137

Post by bionhoward »

droidlyx wrote: post #132
GistNoesis wrote: post #34

I quickly skimmed the paper, got inspired to simplify it, and created some Pytorch Layer : https://github.com/GistNoesis/FourierKAN/ The core is really just a few lines. In the paper they use some spline interpolation to represent 1d function that they sum. Their code seemed aimed at smaller sizes. Instead I chose a different representation, aka fourier coefficients that are used to interpolate the functions of indiv…

Hi Noesis, I just noticed that your implementation, combined with the efficientKAN by Blealtan ( https://github.com/Blealtan/efficient-kan ), results in a structure very similar to Siren(MLP with Sin activations). efficientKAN first computes the common basis functions for all the edge activations and the output can be calculated with a linear combination of the basis. If the basis functions are fourier, then a KAN la…

How could this help us understand the difference between the learned parameters and their gradients? Can the gradients become one with the parameters a la exponential function?

Re: Kolmogorov-Arnold Networks

#138

Post by thesz »

mipt98 wrote: post #134

A more elaborate implementation of this was published years ago, and it wasn't the very first one https://www.science.org/doi/10.1126/science.1165893

https://arxiv.org/abs/1210.7273

  In the article "Distilling free-form natural laws from experimental data", Schmidt and Lipson introduced the idea that free-form natural laws can be learned from experimental measurements in a physical system using symbolic (genetic) regression algorithms. An important claim in this work is that the algorithm finds laws in data without having incorporated any prior knowledge of physics. Upon close inspection, however, we show that their method implicitly incorporates Hamilton's equations of motions and Newton's second law, demystifying how they are able to find Hamiltonians and special classes of Lagrangians from data.
I think this is hilarious.

I cannot get a PDF of your article and instead I will read a commentary on it which appears to be very interesting.

Re: Kolmogorov-Arnold Networks

#139

Post by thesz »

bart1ett wrote: post #46
diwank wrote: post #5

It’d be really cool to see a transformer with the MLP layers swapped for KANs and then compare its scaling properties with vanilla transformers

After trying this out with the fourier implementation above, swapping MLP/Attention Linear layers for KANs (all, or even a few layers) produces diverging loss. KANs don't require normalization for good forward pass dynamics, but may be trickier to train in a deep net.

Note that KANs use LBFGS, which is second-order optimization method. My experience with the use of second-order features suggests that simple gradient descent often leads to divergence.

Re: Kolmogorov-Arnold Networks

#140

Post by renonce »

So a new type of neural network that has been proven to work well on regression tasks common in physics? And tested in practice to fit well on elementary algebra and compositions of complex functions. But no evidence at all if it works on the most basic machine learning tasks like MNIST, not to mention language models.

I mean it's great but at the current state it seems better suited for tasks where an explicit formula exists (though not known) and the goal is to predict it on unknown points (and be able to interpret the formula as a side effect). Deep learning tasks are more of a statistical nature (think models with a cross entropy loss - it's statistically predicting the frequency of different choices of the class/next token), it requires a specialized training procedure and it is designed to fit 100% rather than somewhat close (think linear algebra - it won't be good at it). It would very likely take a radically different idea to apply it to deep learning tasks. The recently updated "Author's note" also mentions this: "KANs are designed for applications where one cares about high accuracy and/or interpretability."

It's great but let's be patient before we see this improve LLM accuracy or be used elsewhere.

Post reply on HN