Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kolmogorov-Arnold Networks

github.com

Post reply on HN
Print view View original

81–90 of 149 posts

Re: Kolmogorov-Arnold Networks

#81

Post by godelski »

abhgh wrote: post #76
godelski wrote: post #48

Earlier quoted context omitted.

There's a ton actually. Just they tend to go through extra rounds of review (or never make it...) and never make it to HN unless there's special circumstances (this one is MIT and CIT). Unfortunately we've let PR become a very powerful force (it's always been a thing, but seems more influential now). We can fight against this by up voting things like this and if you're a reviewee, not focusing on sota (it's clearly b…

Yes seconding this. If you want a broad view of ML IMHO the best places to look at are conference proceedings. The typical review process is imperfect so that still doesn't show you all the interesting work out there (which you mention), but it is still a start wrt diversity of research. I follow LLMs closely but then going through proceedings means I come across exciting research like these [1],[2],[3]. References:…

Honestly, these days I just rely on arxiv. The conferences are so noisy that it is hard to really tell what's useful and what's crap. Twitter is a bit better but still a crap shoot. So as far as it seems to me, there's no real good signal to use to differentiate. And what's the point of journals/conferences if not to provide some reasonable signal? If it is a slot machine, it is useless.

And I feel like we're far too dismissive of instances we see where good papers get rejected. We're too dismissive of the collusion rings. What am I putting in all this time to write and all this time to review (and be an emergency reviewer) if we aren't going to take some basic steps forward? Fuck, I've saved a Welling paper from rejection from two reviewers who admitted to not knowing PDEs, and this was a workshop (should have been accepted into the main conference). I think review works for those already successful, who can p̶a̶y̶ "perform more experiments when requested" their way out of review hell, but we're ignoring a lot of good work simply for lack of m̶o̶n̶e̶y̶ compute. It slows down our progress to reach AGI.

Re: Kolmogorov-Arnold Networks

#82

Post by abhgh »

godelski wrote: post #81
abhgh wrote: post #76

Earlier quoted context omitted.

Yes seconding this. If you want a broad view of ML IMHO the best places to look at are conference proceedings. The typical review process is imperfect so that still doesn't show you all the interesting work out there (which you mention), but it is still a start wrt diversity of research. I follow LLMs closely but then going through proceedings means I come across exciting research like these [1],[2],[3]. References:…

Honestly, these days I just rely on arxiv. The conferences are so noisy that it is hard to really tell what's useful and what's crap. Twitter is a bit better but still a crap shoot. So as far as it seems to me, there's no real good signal to use to differentiate. And what's the point of journals/conferences if not to provide some reasonable signal? If it is a slot machine, it is useless. And I feel like we're far too…

Yes arxiv is a good first source too. I mentioned conferences as a way to get exposed to diversity, but not necessarily (sadly) merit. It has been my experience as an author and reviewer both that review quality has plummeted over the years for the most part. As a reviewer I had to struggle with the ills of "commission and omission" both, i.e., (a) convince other reviewers to see an idea (from a trendy area such as in-context learning) as not novel (because it has been done before, even in the area of LLMs), and (b) see an idea as novel, which wouldn't haven't seemed so initially because some reviewers weren't aware of the background or impact of anything non-LLM, or god forbid, non-DL. As an author this has personally affected me because I had to work on my PhD remotely, so I didn't have access to a bunch of compute and I deliberately picked a non-DL area, and I had to pay the price for that in terms of multiple rejections, reviewer ghosting, journals not responding for years (yes, years).

Re: Kolmogorov-Arnold Networks

#83

Post by godelski »

abhgh wrote: post #82
godelski wrote: post #81

Earlier quoted context omitted.

Honestly, these days I just rely on arxiv. The conferences are so noisy that it is hard to really tell what's useful and what's crap. Twitter is a bit better but still a crap shoot. So as far as it seems to me, there's no real good signal to use to differentiate. And what's the point of journals/conferences if not to provide some reasonable signal? If it is a slot machine, it is useless. And I feel like we're far too…

Yes arxiv is a good first source too. I mentioned conferences as a way to get exposed to diversity, but not necessarily (sadly) merit. It has been my experience as an author and reviewer both that review quality has plummeted over the years for the most part. As a reviewer I had to struggle with the ills of "commission and omission" both, i.e., (a) convince other reviewers to see an idea (from a trendy area such as i…

I've stopped considering novelty at all. The only thing I now consider is if the precise technique has been done before. If not, well I've seen pretty small things change results dramatically. The pattern I've seen that scares me more is that when authors do find simple but effective changes, they end up convoluting the ideas because simplicity and clarity is often confused with novelty. And honestly, revisiting ideas is useful as our environments change. So I don't want to discourage this type of work.

Personally, this has affected me as a late PhD student. Late in the literal sense as I'm not getting my work pushed out (even some SOTA stuff) because of factors like these and my department insists something is wrong with me but will not read my papers, the reviews, or suggest what I need to do besides "publish more." (Literally told to me, "try publishing 5 papers a year, one should get in.") You'll laugh at this, I pushed a paper into a workshop and a major complaint was that I didn't give enough background on StyleGAN because "not everyone would be familiar with the architecture." (while I can understand the comment, 8 pages is not much room when you gotta show pictures on several datasets. My appendix was quite lengthy and included all requested information). We just used a GAN as a proxy because diffusion is much more expensive to train (most common complaints are "not enough datasets" and "how's it scale"). I think this is the reason so many universities use pretrained networks instead of training things from scratch, which just railroads research.

(I also got a paper double desk rejected. First because it was "already published." Took a 2 months for them to realize it was arxiv only. Then they fixed that and rejected again because "didn't cite relevant works" with no mention of what those works were... I've obviously lost all faith in the review process)

Re: Kolmogorov-Arnold Networks

#84

Post by tripplyons »

Lichtso wrote: post #62

1. Interestingly the foundations of this approach and MLP were invented / discovered around the same time about 66 years ago: 1957: https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Arnold_repr... 1958: https://en.wikipedia.org/wiki/Multilayer_perceptron 2. Another advantage of this approach is that it has only one class of parameters (the coefficients of the local activation functions) as opposed to MLP which has thr…

To your 3rd point, most diffusion models already use a transformer-based architecture (U-Net with self attention and cross attention, Vision Transformer, Diffusion Transformer, etc.).

Re: Kolmogorov-Arnold Networks

#85

Post by tripplyons »

kolinko wrote: post #75
Lichtso wrote: post #62

1. Interestingly the foundations of this approach and MLP were invented / discovered around the same time about 66 years ago: 1957: https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Arnold_repr... 1958: https://en.wikipedia.org/wiki/Multilayer_perceptron 2. Another advantage of this approach is that it has only one class of parameters (the coefficients of the local activation functions) as opposed to MLP which has thr…

I may be wrong but with midern llms biases aren’t really used any more.

From what I remember, larger LLMs like PaLM don't use biases for training stability, but smaller ones tend to still use them.

Re: Kolmogorov-Arnold Networks

#86

Post by londons_explore »

krasin wrote: post #12

I've spent some time playing with their Jupyter notebooks. The most useful (to me, anyway) is their Example_3_classfication.ipynb ([1]). It works as advertised with the parameters selected by the authors, but if we modified the network shape in the second half of the tutorial (Classification formulation) from (2, 2) to (2, 2, 2), it fails to generalize. The training loss gets down to 1e-9, while test loss stays aroun…

> I would really like to see a bigger example

This. I don't think toy examples are useful for modern ML techniques. If you tested big ideas in ML (transformers, LSTM's, ADAM) on a training dataset of 50 numbers trying to fit a y=sin(x) curve, I think you'd wrongly throw these ideas out.

Re: Kolmogorov-Arnold Networks

#87

Post by nyrikki »

ubj wrote: post #36

Very interesting! Kolmogorov neutral networks can represent discontinuous functions [1], but I've wondered about how practically applicable they are. This repo seems to show that they have some use after all. [1]: https://arxiv.org/abs/2311.00049

Not for discontinuous functions, as your paper explains, we know that g exists for discontinuous bounded, but nothing to find it with.

> A practical construction of g in cases with discontinuous bounded and un- bounded functions is not yet known. For such cases Theorem 2.1 gives only a theoretical understanding of the representation problem. This is because for the representation of discontinuous bounded functions we have derived (2.1) from the fact that the range of the operator Z∗ is the whole space of bounded functions B(Id). This fact directly gives us a formula (2.1) but does not tell how the bounded one-variable function g is attained. For the representation of unbounded functions we have used a linear extension of the functional F , existence of which is based on Zorn’s lemma (see, e.g., [19, Ch. 3]). Application of Zorn’s lemma provides no mechanism for practical construction of such an extension. Zorn’s lemma helps to assert only its existence.

If you look at the OP post arxiv link, you will see they are using splines .

https://arxiv.org/abs/2404.19756

Still interesting and potentially useful, but not useful for discontinuous functions without further discoveries.

If I am wrong please provide a link, it is of great interest to me.

Re: Kolmogorov-Arnold Networks

#88

Post by cloudhan »

agnosticmantis wrote: post #71
GistNoesis wrote: post #34

I quickly skimmed the paper, got inspired to simplify it, and created some Pytorch Layer : https://github.com/GistNoesis/FourierKAN/ The core is really just a few lines. In the paper they use some spline interpolation to represent 1d function that they sum. Their code seemed aimed at smaller sizes. Instead I chose a different representation, aka fourier coefficients that are used to interpolate the functions of indiv…

How GPU-friendly is this class of models?

Very unfriendly.

The symbolic library (type of activations) requires a branching at the very core of the kernel. GPU will need to serialized on these operations warp-wise.

To optimize, you might want to do a scan operation beforehand and dispatch to activation funcs in a warp specialized way, this, however, makes the global memory read/write non-coalesced.

You then may sort the input based on type of activations and store it in that order, this makes the gmem IO coalesced but requires gather and scatter as pre and post processing.

Re: Kolmogorov-Arnold Networks

#89

Post by versteegen »

godelski wrote: post #81
abhgh wrote: post #76

Earlier quoted context omitted.

Yes seconding this. If you want a broad view of ML IMHO the best places to look at are conference proceedings. The typical review process is imperfect so that still doesn't show you all the interesting work out there (which you mention), but it is still a start wrt diversity of research. I follow LLMs closely but then going through proceedings means I come across exciting research like these [1],[2],[3]. References:…

Honestly, these days I just rely on arxiv. The conferences are so noisy that it is hard to really tell what's useful and what's crap. Twitter is a bit better but still a crap shoot. So as far as it seems to me, there's no real good signal to use to differentiate. And what's the point of journals/conferences if not to provide some reasonable signal? If it is a slot machine, it is useless. And I feel like we're far too…

> I've saved a Welling paper from rejection from two reviewers who admitted to not knowing PDEs

Thank you for fighting the good fight.

This is why I love OpenReview, I can spot and ignore nonsensical reviewer criticisms and ratings and look for the insightful comments and rebuttals. Many reviewers do put in a lot of very valuable work reading and critiquing most of which would go to waste if not made public.

Post reply on HN