Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kolmogorov-Arnold Networks

github.com

Post reply on HN
Print view View original

101–110 of 149 posts

Re: Kolmogorov-Arnold Networks

#101

Post by godelski »

pama wrote: post #96
godelski wrote: post #83

Earlier quoted context omitted.

I've stopped considering novelty at all. The only thing I now consider is if the precise technique has been done before. If not, well I've seen pretty small things change results dramatically. The pattern I've seen that scares me more is that when authors do find simple but effective changes, they end up convoluting the ideas because simplicity and clarity is often confused with novelty. And honestly, revisiting idea…

Sorry to hear all this (after writing my other sibling comment). Please don’t lose faith in the review process. It is still useful. Until the AGI can be better reviewers, which is hopefully not too far in the future.

For me to regain faith in the review process I need to actually see some semblance of the review process working.

So far, instead, I've seen:

  - Banning social media posting so that only big tech and collusion positing can happen to "protect the little guy"
  - Undoing the ban to lots of complaints
  - Instituting a no LLM policy with no teeth and no method to actually verify
  - Instituting a high school track to get those rich kids in sooner
Until I see such changes like "we're going to focus on review quality" I'm going to continue thinking it is a scam. They get paid by my tax dollars, by private companies, and I volunteer time, for what...? Something a LLM could have actually done better? I'm seeing great papers from big (and small) labs get turned down while terrible papers are getting accepted. Collusion rings go unpunished. And methods get more and more convoluted as everyone tries to game the system.

You think of all people, we, ML, would understand reward hacking. But until we admit it, we can't solve it. And if we can't solve it here, how the hell are we going to convince anyone we're going to create safe AGI?

Re: Kolmogorov-Arnold Networks

#102

Post by [deleted] »

versteegen wrote: post #89
godelski wrote: post #81

Earlier quoted context omitted.

Honestly, these days I just rely on arxiv. The conferences are so noisy that it is hard to really tell what's useful and what's crap. Twitter is a bit better but still a crap shoot. So as far as it seems to me, there's no real good signal to use to differentiate. And what's the point of journals/conferences if not to provide some reasonable signal? If it is a slot machine, it is useless. And I feel like we're far too…

> I've saved a Welling paper from rejection from two reviewers who admitted to not knowing PDEs Thank you for fighting the good fight. This is why I love OpenReview, I can spot and ignore nonsensical reviewer criticisms and ratings and look for the insightful comments and rebuttals. Many reviewers do put in a lot of very valuable work reading and critiquing most of which would go to waste if not made public.

[deleted]

Re: Kolmogorov-Arnold Networks

#103

Post by godelski »

abhgh wrote: post #98
godelski wrote: post #83

Earlier quoted context omitted.

I've stopped considering novelty at all. The only thing I now consider is if the precise technique has been done before. If not, well I've seen pretty small things change results dramatically. The pattern I've seen that scares me more is that when authors do find simple but effective changes, they end up convoluting the ideas because simplicity and clarity is often confused with novelty. And honestly, revisiting idea…

Sorry to hear that. My experiences haven't been very different. I really can't tell if the current review process is the least bad among alternatives or is there something better (if so, what is it?).

I'm sorry to hear that too. I really wish there was something that could be done. I imagine a lot of graduate students are in complicated situations because of this.

As for alternatives: I don't see why we don't just push to OpenReview and call it a day. We can link our code, it has revisions, and people can comment and review. I don't see what the advantage of having 1-3 referees who don't want to read my paper and have no interest in it but have strong incentives to reject it is any meaningful signal of value. I'll take arxiv over their opinions.

Re: Kolmogorov-Arnold Networks

#104

Post by godelski »

versteegen wrote: post #89
godelski wrote: post #81

Earlier quoted context omitted.

Honestly, these days I just rely on arxiv. The conferences are so noisy that it is hard to really tell what's useful and what's crap. Twitter is a bit better but still a crap shoot. So as far as it seems to me, there's no real good signal to use to differentiate. And what's the point of journals/conferences if not to provide some reasonable signal? If it is a slot machine, it is useless. And I feel like we're far too…

> I've saved a Welling paper from rejection from two reviewers who admitted to not knowing PDEs Thank you for fighting the good fight. This is why I love OpenReview, I can spot and ignore nonsensical reviewer criticisms and ratings and look for the insightful comments and rebuttals. Many reviewers do put in a lot of very valuable work reading and critiquing most of which would go to waste if not made public.

I like OR too and I wish we would just post to there instead. It has everything we need, and I see no value from the venues. No one wants to act in good faith and they have every incentive not to.

And I gotta say, I'm not going to put up a fight much longer. As soon as I get out of my PhD I intend to just post to OR.

Re: Kolmogorov-Arnold Networks

#105

Post by samus »

jiggawatts wrote: post #100
cloudhan wrote: post #88

Earlier quoted context omitted.

Very unfriendly. The symbolic library (type of activations) requires a branching at the very core of the kernel. GPU will need to serialized on these operations warp-wise. To optimize, you might want to do a scan operation beforehand and dispatch to activation funcs in a warp specialized way, this, however, makes the global memory read/write non-coalesced. You then may sort the input based on type of activations and…

Wouldn't it be faster to calculate every function type and then just multiply them by 0s or 1s to keep the active ones?

That's pretty much how branching on GPUs already works.

Re: Kolmogorov-Arnold Networks

#106

Post by sevagh »

GistNoesis wrote: post #34

I quickly skimmed the paper, got inspired to simplify it, and created some Pytorch Layer : https://github.com/GistNoesis/FourierKAN/ The core is really just a few lines. In the paper they use some spline interpolation to represent 1d function that they sum. Their code seemed aimed at smaller sizes. Instead I chose a different representation, aka fourier coefficients that are used to interpolate the functions of indiv…

Does your code work? Did you train it? Any graphs?

>Of course, if my code doesn't work, it doesn't mean theirs doesn't.

But, _does_ it work?

Re: Kolmogorov-Arnold Networks

#107

Post by sevagh »

godelski wrote: post #48
cs702 wrote: post #39

It's so refreshing to come across new AI research different from the usual "we modified a transformer in this and that way and got slightly better results on this and that benchmark." All those new papers proposing incremental improvements are important, but... everyone is getting a bit tired of them. Also, anecdotal evidence and recent work suggest we're starting to run into fundamental limits inherent to transforme…

There's a ton actually. Just they tend to go through extra rounds of review (or never make it...) and never make it to HN unless there's special circumstances (this one is MIT and CIT). Unfortunately we've let PR become a very powerful force (it's always been a thing, but seems more influential now). We can fight against this by up voting things like this and if you're a reviewee, not focusing on sota (it's clearly b…

For example, I find Spike Neural Networks to be cool, but until they reach SOTA, how can they displace conventional neural networks?

Re: Kolmogorov-Arnold Networks

#108

Post by godelski »

pama wrote: post #94
godelski wrote: post #81

Earlier quoted context omitted.

Honestly, these days I just rely on arxiv. The conferences are so noisy that it is hard to really tell what's useful and what's crap. Twitter is a bit better but still a crap shoot. So as far as it seems to me, there's no real good signal to use to differentiate. And what's the point of journals/conferences if not to provide some reasonable signal? If it is a slot machine, it is useless. And I feel like we're far too…

I agree with almost all you said except that Twitter is better than top conferences, and I take a contrarian view that reviewers slow down AGI with requests for additional experiments. Without going into specifics, which you can probably guess based on your background, too many ideas that work well, even optimally, at small scale fail horribly at large scale. Other ideas that work at super specialized settings don’t…

> too many ideas that work well, even optimally, at small scale fail horribly at large scale.

Not that I disagree, but I don't think that's a reason to not publish. There's another way to rephrase what you've said

  many ideas that work well at small scales do not trivially work at large scales
But this is true for many works, even transformers. You don't just scale by turning up model parameters and data. You can, but generally more things are going on. So why hold these works back because of that? There may be nuggets in there that may be of value and people may learn how to scale them. Just because they don't scale (now or ever) doesn't mean they aren't of value (and let's be honest, if they don't scale, this is a real killer for the "scale is all you need" people)

> Other ideas that work at super specialized settings don’t transfer or don’t generalize.

It is also hard to tell if these are hyper-parameter settings. Not that I disagree with you, but it is hard to tell.

> Correlations in huge multimodal datasets are way more complicated than most humans can grasp and we will not get to AGI before we can have a large enough group of people dealing with such data routinely.

I'm not sure I understand your argument here. The people I know that work at scale often have the worst understanding of large data. Not understanding the differences between density in a normal distribution and a uniform. Thinking that LERPing in a normal yields representative data. Or cosine simularity and orthogonality. IME people that work at scale benefit from being able to throw compute at problems.

> we don’t do anybody a favor by increasing the entropy of the publications in the huge ML conferences

You and I have very different ideas as to what constitutes information gain. I would say a majority of people studying two models (LLMs and diffusion) results in lower gain, not more.

And as I've said above, I don't care about novelty. It's a meaningless term. (and I wish to god people would read the fucking conference reviewer guidelines as they constantly violate them when discussing novelty)

Re: Kolmogorov-Arnold Networks

#109

Post by vindex10 »

coderenegade wrote: post #92

I was under the impression that graph neural nets already trained learnable functions on graph edges rather than nodes, albeit typically on a fully connected graph. Is there any comparison to just a basic GNN here?

Bayesian networks learn probability functions, but looks like only their tabulated versions:

https://en.wikipedia.org/wiki/Bayesian_network#Graphical_mod...

> Each node is associated with a probability function that takes, as input, a particular set of values for the node's parent variables, and gives (as output) the probability (or probability distribution, if applicable) of the variable represented by the node. For example, if m parent nodes represent m Boolean variables, then the probability function could be represented by a table of 2^m entries, one entry for each of the 2^m possible parent combinations.

Re: Kolmogorov-Arnold Networks

#110

Post by samus »

krasin wrote: post #12

I've spent some time playing with their Jupyter notebooks. The most useful (to me, anyway) is their Example_3_classfication.ipynb ([1]). It works as advertised with the parameters selected by the authors, but if we modified the network shape in the second half of the tutorial (Classification formulation) from (2, 2) to (2, 2, 2), it fails to generalize. The training loss gets down to 1e-9, while test loss stays aroun…

It's possible to run it on CUDA. One of their examples shows how. But I found it's slower than on CPU. I'm actually not really surprised since running something on GPU is not a guarantee that it's gonna be fast, especially when lots of branching is involved.

Unfortunately, I had to modify KAN.py and KANLayer.py to make it work as not all relevant tensor are put on the correct device. In some places the formatting even suggests that there was previously a device argument.

Post reply on HN