Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kolmogorov-Arnold Networks

github.com

Post reply on HN
Print view View original

31–40 of 149 posts

Re: Kolmogorov-Arnold Networks

#31

Post by verdverm »

gxyt6gfy5t wrote: post #29
thom wrote: post #28

Earlier quoted context omitted.

This makes me wonder what you could achieve if instead of iteratively growing the grid, or worrying about pruning or regularization, you governed network topology with some sort of evolutionary algorithm.

Believe there is a Google paper out there that tried that

1000s, there is a whole field and set of conferences. You can find more by searching "Genetic Programming" or "Symbolic Regression"

KAN, with the library of variables and math operators, very much resembles this family of algos, problems, and limitations. The lowest hanging fruit they usually leave on the proverbial tree is that you can use fast regression techniques for the constants and coefficients. No need to leave it up to random perturbations or gradient descent. What you really need to figure out is the form or shape of the model, rather than leaving it up to the human (in KAN)

Re: Kolmogorov-Arnold Networks

#32

Post by xpe »

cbsmith wrote: post #6

Feels like someone stuffed splines into decision trees.

splines, yes.

I'm not seeing decision trees, though. Am I missing something?

> "KANs’ nodes simply sum incoming signals without applying any non-linearities." (page 2 of the PDF)

Re: Kolmogorov-Arnold Networks

#33

Post by galangalalgol »

wongarsu wrote: post #27
mxwsn wrote: post #11

From the preprint - 100 input dimensions is considered "high", and most problems considered have 5 or fewer input dimensions. This is typical of physics-inspired settings I've seen considered in ML. The next step would be demonstrating them on MNIST, which, at 784 dimensions is tiny by modern standards.

In actual business processes there are lots of ML problems with fewer than 100 input dimensions. But for most of them decision trees are still competitive with neural networks or even outperform them.

The aid to explainability seems at least somewhat compelling. Understanding what a random forest did isn't always easy. And if what you want isn't the model but the closed form of what the model does, this could be quite useful. When those hundred input dimensions interact nonlinearly in a million ways thats nice. Or more likely I'd use it when I don't want to find a pencil to derive the closed form of what I'm trying to do.

Re: Kolmogorov-Arnold Networks

#34

Post by GistNoesis »

I quickly skimmed the paper, got inspired to simplify it, and created some Pytorch Layer :

https://github.com/GistNoesis/FourierKAN/

The core is really just a few lines.

In the paper they use some spline interpolation to represent 1d function that they sum. Their code seemed aimed at smaller sizes. Instead I chose a different representation, aka fourier coefficients that are used to interpolate the functions of individual coordinates.

It should give an idea of Kolmogorov-Arnold networks representation power, it should probably converge easier than their spline version but spline version have less operations.

Of course, if my code doesn't work, it doesn't mean theirs doesn't.

Feel free to experiment and publish paper if you want.

Re: Kolmogorov-Arnold Networks

#35

Post by lccerina »

vladms wrote: post #23
cess11 wrote: post #13

Earlier quoted context omitted.

"Technical progress" has been destroying our habitat for centuries, causing lots of other species to go extinct. Pretty much the entire planet surface has been 'technically progressed', spreading plastics, climate change and whatnot over the entirety of it. Are you assuming that this particular "progress" would be relatively innocent?

Many species went extinct during Earth's history. Evolution requires quite aggressive competition. The way the habitat got destroyed by humans is stupid because it might put us in danger. You can call me "speciesist" but I do care more for humans rather than for a particular other specie. So I think progress should be geared towards human species survival and if possible preventing other species extinction. Some of t…

If other species die, we follow shortly. This anthropocentric view really ignore how much of our food chain exists because of other animals surviving despite human activities.

Re: Kolmogorov-Arnold Networks

#37

Post by brrrrrm »

mp187 wrote: post #22
diwank wrote: post #5

It’d be really cool to see a transformer with the MLP layers swapped for KANs and then compare its scaling properties with vanilla transformers

Why was this your first thought? Is a limiting factor to transformers the MLP layer? I thought the bottleneck was in the renormalization part.

At small input size, yes the MLP dominates compute. At large input attention matters more

Re: Kolmogorov-Arnold Networks

#39

Post by cs702 »

It's so refreshing to come across new AI research different from the usual "we modified a transformer in this and that way and got slightly better results on this and that benchmark." All those new papers proposing incremental improvements are important, but... everyone is getting a bit tired of them. Also, anecdotal evidence and recent work suggest we're starting to run into fundamental limits inherent to transformers, so we may well need new alternatives.[a]

The best thing about this new work is that it's not an either/or proposition. The proposed "learnable spline interpolations as activation functions" can be used in conventional DNNs, to improve their expressivity. Now we just have to test the stuff to see if it really works better.

Very nice. Thank you for sharing this work here!

---

[a] https://news.ycombinator.com/item?id=40179232

Re: Kolmogorov-Arnold Networks

#40

Post by glebnovikov »

cs702 wrote: post #39

It's so refreshing to come across new AI research different from the usual "we modified a transformer in this and that way and got slightly better results on this and that benchmark." All those new papers proposing incremental improvements are important, but... everyone is getting a bit tired of them. Also, anecdotal evidence and recent work suggest we're starting to run into fundamental limits inherent to transforme…

> Everyone is getting tired of those papers.

This is science as is :)

95% percent will produce mediocre-to-nice improvements to what we already have so there were reserachers that eventually grow up and do something really exciting

Post reply on HN