Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kolmogorov-Arnold Networks

github.com

Post reply on HN
Print view View original

51–60 of 149 posts

Re: Kolmogorov-Arnold Networks

#51

Post by [deleted] »

glebnovikov wrote: post #40
cs702 wrote: post #39

It's so refreshing to come across new AI research different from the usual "we modified a transformer in this and that way and got slightly better results on this and that benchmark." All those new papers proposing incremental improvements are important, but... everyone is getting a bit tired of them. Also, anecdotal evidence and recent work suggest we're starting to run into fundamental limits inherent to transforme…

> Everyone is getting tired of those papers. This is science as is :) 95% percent will produce mediocre-to-nice improvements to what we already have so there were reserachers that eventually grow up and do something really exciting

[deleted]

Re: Kolmogorov-Arnold Networks

#52

Post by [deleted] »

[deleted] wrote: post #51
glebnovikov wrote: post #40

Earlier quoted context omitted.

> Everyone is getting tired of those papers. This is science as is :) 95% percent will produce mediocre-to-nice improvements to what we already have so there were reserachers that eventually grow up and do something really exciting

[deleted]

[deleted]

Re: Kolmogorov-Arnold Networks

#53

Post by Lichtso »

reynoldss wrote: post #38

Perhaps a hasty comment but linear combinations of B-splines are yet another (higher-degree) B-spline. Isn't this simply fitting high degree B-splines to functions?

That would be true for a single node / single layer. But once the output of one layer is fed into the input of the next it is not just a linear combination of splines anymore.

Re: Kolmogorov-Arnold Networks

#54

Post by beagle3 »

cs702 wrote: post #39

It's so refreshing to come across new AI research different from the usual "we modified a transformer in this and that way and got slightly better results on this and that benchmark." All those new papers proposing incremental improvements are important, but... everyone is getting a bit tired of them. Also, anecdotal evidence and recent work suggest we're starting to run into fundamental limits inherent to transforme…

I read a book on NNs by Robert Hecht Nielsen in 1989, during the NN hype of the time (I believe it was the 2nd hype cycle, the first beginning with Rosenblatt’s original hardware perceptron and dying with Minsky and Pappert’s “Perceptrons” manuscript a decade or two earlier).

Everything described was laughably basic by modern standards, but the motivation given in that book was the Kolmogorov representation theorem: a modest 3 layer networks with the right activation function can represent any continuous m-to-n function.

Most research back then focused on 3 layer networks, possibly for that reason. Sigmoid activation was king, and vanishing gradients the main issue. It took 2 decades until AlexNet brought NN research back from the AI winter of the 1990’s

Re: Kolmogorov-Arnold Networks

#55

Post by cbsmith »

xpe wrote: post #32
cbsmith wrote: post #6

Feels like someone stuffed splines into decision trees.

splines, yes. I'm not seeing decision trees, though. Am I missing something? > "KANs’ nodes simply sum incoming signals without applying any non-linearities." (page 2 of the PDF)

I definitely think I'm projecting and maybe seeing things that aren't there. If you replaced splines with linear weights, it kind of looks like a decision tree to me.

Re: Kolmogorov-Arnold Networks

#56

Post by beagle3 »

yobbo wrote: post #26

https://kindxiaoming.github.io/pykan/intro.html At the end of this example, they recover the symbolic formula that generated their training set: exp(x₂² + sin(3.14x₁)). It's like a computation graph with a library of "activation functions" that is optimised, and then pruned. You can recover good symbolic formulas from the pruned graph. Maybe not meaningful for MNIST.

I wonder if Breiman’s ACE (alternating conditional expectation) is useful as a building block here.

It will easily recover this formula, because it is separable under the log transformation (which ACE recovers as well).

But ACE doesn’t work well on unseparable problems - not sure how well KAN will.

Re: Kolmogorov-Arnold Networks

#58

Post by godelski »

krasin wrote: post #21
krasin wrote: post #12

I've spent some time playing with their Jupyter notebooks. The most useful (to me, anyway) is their Example_3_classfication.ipynb ([1]). It works as advertised with the parameters selected by the authors, but if we modified the network shape in the second half of the tutorial (Classification formulation) from (2, 2) to (2, 2, 2), it fails to generalize. The training loss gets down to 1e-9, while test loss stays aroun…

Update2: got it to 100% training accuracy, 99% test accuracy with (2, 2, 2) shape. Changes: 1. Increased the training set from 1000 to 100k samples. This solved overfitting. 2. In the dataset generation, slightly reduced noise (0.1 -> 0.07) so that classes don't overlap. With an overlap, naturally, it's impossible to hit 100%. 3. Most important & specific to KANs: train for 30 steps with grid=5 (5 segments for each a…

> Increased the training set from 1000 to 100k samples. This solved overfitting.

Solved over fitting or created more? Even if your sets are completely disjoint with something like two moons the more data you have the lower the variance.

Re: Kolmogorov-Arnold Networks

#59

Post by nextaccountic »

SpaceManNabs wrote: post #45

How does back propagation work now? Do these suffer from vanishing or exploding gradients?

At page 6 it explains how they did back propagation https://arxiv.org/pdf/2404.19756 (and in page 2 it says that previous efforts to leverage Kolmogorov-Arnold representation failed to use backpropagation), so maybe using backpropagation to train multilayer networks with this architecture is their main contribution?

> Unsurprisingly, the possibility of using Kolmogorov-Arnold representation theorem to build neuralnetworks has been studied [8, 9, 10, 11, 12, 13]. However, most work has stuck with the original depth-2 width-(2n + 1) representation, and did not have the chance to leverage more modern techniques (e.g., back propagation) to train the networks. Our contribution lies in generalizing the original Kolmogorov-Arnold representation to arbitrary widths and depths, revitalizing and contextualizing it in today’s deep learning world, as well as using extensive empirical experiments to highlight its potential role as a foundation model for AI + Science due to its accuracy and interpretability.

Re: Kolmogorov-Arnold Networks

#60

Post by snewman »

ALittleLight wrote: post #7

I can't assess this, but I do worry that overnight some algorithmic advance will enhance LLMs by orders of magnitude and the next big model to get trained is suddenly 10,000x better than GPT-4 and nobody's ready for it.

I think this is unlikely. There has never (in the visible fossil record) been a mutation that suddenly made tigers an order of magnitude stronger and faster, or humans an order of magnitude more intelligent. It's been a long time (if ever?) since chip transistor density made a multiple-order-of-magnitude leap. Any complex optimized system has many limiting factors and it's unlikely that all of them would leap forward at once. The current generation of LLMs are not as complex or optimized as tigers or humans, but they're far enough along that changing one thing is unlikely to result in a giant leap.

If and when something radically better comes along, say an alternative to back-propagation that is more like the way our brains learn, it will need a lot of scaling and refinement to catch up with the then-current LLM.

Post reply on HN