Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kolmogorov-Arnold Networks

github.com

Post reply on HN
Print view View original

21–30 of 149 posts

Re: Kolmogorov-Arnold Networks

#21

Post by krasin »

krasin wrote: post #12

I've spent some time playing with their Jupyter notebooks. The most useful (to me, anyway) is their Example_3_classfication.ipynb ([1]). It works as advertised with the parameters selected by the authors, but if we modified the network shape in the second half of the tutorial (Classification formulation) from (2, 2) to (2, 2, 2), it fails to generalize. The training loss gets down to 1e-9, while test loss stays aroun…

Update2: got it to 100% training accuracy, 99% test accuracy with (2, 2, 2) shape.

Changes:

1. Increased the training set from 1000 to 100k samples. This solved overfitting.

2. In the dataset generation, slightly reduced noise (0.1 -> 0.07) so that classes don't overlap. With an overlap, naturally, it's impossible to hit 100%.

3. Most important & specific to KANs: train for 30 steps with grid=5 (5 segments for each activation function), then 30 steps with grid=10 (and initializing from the previous model), and then 30 steps with grid=20. This is idiomatic to KANs and covered in the Example_1_function_fitting.ipynb: https://github.com/KindXiaoming/pykan/blob/master/tutorials/...

Overall, my impressions are:

- it works!

- the reference implementation is very slow. A GPU implementation is dearly needed.

- it feels like it's a bit too non-linear and training is not as stable as it's with MLP + ReLU.

- Scaling is not guaranteed to work well. Really need to see if MNIST is possible to solve with this approach.

I will definitely keep an eye on this development.

Re: Kolmogorov-Arnold Networks

#22

Post by mp187 »

diwank wrote: post #5

It’d be really cool to see a transformer with the MLP layers swapped for KANs and then compare its scaling properties with vanilla transformers

Why was this your first thought? Is a limiting factor to transformers the MLP layer? I thought the bottleneck was in the renormalization part.

Re: Kolmogorov-Arnold Networks

#23

Post by vladms »

cess11 wrote: post #13
6mian wrote: post #10

Earlier quoted context omitted.

What to be worried about? Technical progress will happen, sometimes by sudden jumps. Some company will become a leader, competitors will catch up after a while.

"Technical progress" has been destroying our habitat for centuries, causing lots of other species to go extinct. Pretty much the entire planet surface has been 'technically progressed', spreading plastics, climate change and whatnot over the entirety of it. Are you assuming that this particular "progress" would be relatively innocent?

Many species went extinct during Earth's history. Evolution requires quite aggressive competition.

The way the habitat got destroyed by humans is stupid because it might put us in danger. You can call me "speciesist" but I do care more for humans rather than for a particular other specie.

So I think progress should be geared towards human species survival and if possible preventing other species extinction. Some of the current developments are a bit too much on the side of "I don't care about anyone's survival" (which is stupid and inefficient).

Re: Kolmogorov-Arnold Networks

#24

Post by 6mian »

cess11 wrote: post #19
6mian wrote: post #14

Earlier quoted context omitted.

On the other hand, the same "technical progress" (if we're putting machine learning, deforestation, and mining in the same bag) gave you medicine, which turns many otherwise deadly diseases into inconveniences and allows you to work less than 12 hrs/7 days per week to not die from hunger in a large portion of the world. A few hundred years ago, unless you were born into the lucky 0.01% of the ruling population, worki…

Where did you learn that history? What do you mean by "better"?

I had a European peasant in the 1600-1700s in mind when I wrote about the amount of work. During the season, they worked all day; off-season, they had "free time" that went into taking care of the household, inventory, etc., so it's still work. Can't quickly find a reliable source in English I could link, so I can be wrong here.

"Better" was referring to what OP wrote in the top comment. I guess 10x faster, 10x longer context, and 100x less prone to hallucinations would make a good "10k x better" than GPT-4.

Re: Kolmogorov-Arnold Networks

#26

Post by yobbo »

https://kindxiaoming.github.io/pykan/intro.html

At the end of this example, they recover the symbolic formula that generated their training set: exp(x₂² + sin(3.14x₁)).

It's like a computation graph with a library of "activation functions" that is optimised, and then pruned. You can recover good symbolic formulas from the pruned graph.

Maybe not meaningful for MNIST.

Re: Kolmogorov-Arnold Networks

#27

Post by wongarsu »

mxwsn wrote: post #11

From the preprint - 100 input dimensions is considered "high", and most problems considered have 5 or fewer input dimensions. This is typical of physics-inspired settings I've seen considered in ML. The next step would be demonstrating them on MNIST, which, at 784 dimensions is tiny by modern standards.

In actual business processes there are lots of ML problems with fewer than 100 input dimensions. But for most of them decision trees are still competitive with neural networks or even outperform them.

Re: Kolmogorov-Arnold Networks

#28

Post by thom »

krasin wrote: post #21
krasin wrote: post #12

I've spent some time playing with their Jupyter notebooks. The most useful (to me, anyway) is their Example_3_classfication.ipynb ([1]). It works as advertised with the parameters selected by the authors, but if we modified the network shape in the second half of the tutorial (Classification formulation) from (2, 2) to (2, 2, 2), it fails to generalize. The training loss gets down to 1e-9, while test loss stays aroun…

Update2: got it to 100% training accuracy, 99% test accuracy with (2, 2, 2) shape. Changes: 1. Increased the training set from 1000 to 100k samples. This solved overfitting. 2. In the dataset generation, slightly reduced noise (0.1 -> 0.07) so that classes don't overlap. With an overlap, naturally, it's impossible to hit 100%. 3. Most important & specific to KANs: train for 30 steps with grid=5 (5 segments for each a…

This makes me wonder what you could achieve if instead of iteratively growing the grid, or worrying about pruning or regularization, you governed network topology with some sort of evolutionary algorithm.

Re: Kolmogorov-Arnold Networks

#29

Post by gxyt6gfy5t »

thom wrote: post #28
krasin wrote: post #21

Earlier quoted context omitted.

Update2: got it to 100% training accuracy, 99% test accuracy with (2, 2, 2) shape. Changes: 1. Increased the training set from 1000 to 100k samples. This solved overfitting. 2. In the dataset generation, slightly reduced noise (0.1 -> 0.07) so that classes don't overlap. With an overlap, naturally, it's impossible to hit 100%. 3. Most important & specific to KANs: train for 30 steps with grid=5 (5 segments for each a…

This makes me wonder what you could achieve if instead of iteratively growing the grid, or worrying about pruning or regularization, you governed network topology with some sort of evolutionary algorithm.

Believe there is a Google paper out there that tried that

Re: Kolmogorov-Arnold Networks

#30

Post by verdverm »

thom wrote: post #28
krasin wrote: post #21

Earlier quoted context omitted.

Update2: got it to 100% training accuracy, 99% test accuracy with (2, 2, 2) shape. Changes: 1. Increased the training set from 1000 to 100k samples. This solved overfitting. 2. In the dataset generation, slightly reduced noise (0.1 -> 0.07) so that classes don't overlap. With an overlap, naturally, it's impossible to hit 100%. 3. Most important & specific to KANs: train for 30 steps with grid=5 (5 segments for each a…

This makes me wonder what you could achieve if instead of iteratively growing the grid, or worrying about pruning or regularization, you governed network topology with some sort of evolutionary algorithm.

You can do much better by growing an AST with memoization and non-linear regression. So much so, the EVO folks gave a best paper to a non-EVO, deterministic algorithm at their conference

https://seminars.math.binghamton.edu/ComboSem/worm-chiu.pge_... (author)

Post reply on HN