71–80 of 149 posts
I quickly skimmed the paper, got inspired to simplify it, and created some Pytorch Layer : https://github.com/GistNoesis/FourierKAN/ The core is really just a few lines. In the paper they use some spline interpolation to represent 1d function that they sum. Their code seemed aimed at smaller sizes. Instead I chose a different representation, aka fourier coefficients that are used to interpolate the functions of indiv…
1. Interestingly the foundations of this approach and MLP were invented / discovered around the same time about 66 years ago: 1957: https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Arnold_repr... 1958: https://en.wikipedia.org/wiki/Multilayer_perceptron 2. Another advantage of this approach is that it has only one class of parameters (the coefficients of the local activation functions) as opposed to MLP which has thr…
Biases are just weights on an always on input. There isn't much difference between weights of a linear sum and coefficients of a spline.
Granted, however this approach does not require that constant-one input either.
> There isn't much difference between weights of a linear sum and coefficients of a function.
Yes, the trained function coefficients of this approach are the equivalent to the trained weights of MLP. Still this approach does not require the globally uniform activation function of MLP.
It's so refreshing to come across new AI research different from the usual "we modified a transformer in this and that way and got slightly better results on this and that benchmark." All those new papers proposing incremental improvements are important, but... everyone is getting a bit tired of them. Also, anecdotal evidence and recent work suggest we're starting to run into fundamental limits inherent to transforme…
There's a ton actually. Just they tend to go through extra rounds of review (or never make it...) and never make it to HN unless there's special circumstances (this one is MIT and CIT). Unfortunately we've let PR become a very powerful force (it's always been a thing, but seems more influential now). We can fight against this by up voting things like this and if you're a reviewee, not focusing on sota (it's clearly b…
Yes, I agree. The two most common patterns I've noticed in research that does show up on HN are: 1) It outright improves, or has the potential to improve, applications currently used in production by many HN readers. In other words, it's not just navel-gazing. 2) The authors and/or their organizations are well-known, as you suggest.
Earlier quoted context omitted.
Biases are just weights on an always on input. There isn't much difference between weights of a linear sum and coefficients of a spline.
> Biases are just weights on an always on input. Granted, however this approach does not require that constant-one input either. > There isn't much difference between weights of a linear sum and coefficients of a function. Yes, the trained function coefficients of this approach are the equivalent to the trained weights of MLP. Still this approach does not require the globally uniform activation function of MLP.
The only question is if splines are more efficient than lines at describing general functions at the billion to trillion parameter count.
1. Interestingly the foundations of this approach and MLP were invented / discovered around the same time about 66 years ago: 1957: https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Arnold_repr... 1958: https://en.wikipedia.org/wiki/Multilayer_perceptron 2. Another advantage of this approach is that it has only one class of parameters (the coefficients of the local activation functions) as opposed to MLP which has thr…
It's so refreshing to come across new AI research different from the usual "we modified a transformer in this and that way and got slightly better results on this and that benchmark." All those new papers proposing incremental improvements are important, but... everyone is getting a bit tired of them. Also, anecdotal evidence and recent work suggest we're starting to run into fundamental limits inherent to transforme…
There's a ton actually. Just they tend to go through extra rounds of review (or never make it...) and never make it to HN unless there's special circumstances (this one is MIT and CIT). Unfortunately we've let PR become a very powerful force (it's always been a thing, but seems more influential now). We can fight against this by up voting things like this and if you're a reviewee, not focusing on sota (it's clearly b…
References:
[1] A grad.-based way to optimize axis-parallel and oblique decision trees: the Tree Alternating Optimization (TAO) algorithm https://proceedings.neurips.cc/paper_files/paper/2018/file/1.... An extension was the softmax tree https://aclanthology.org/2021.emnlp-main.838/.
[2] XAI explains models, but can you recommend corrective actions? FACE: feasible and Actionable Counterfactual Explanations https://arxiv.org/pdf/1909.09369, Algorithmic Recourse: from Counterfactual Explanations to Interventions https://arxiv.org/pdf/2002.06278
[3] OBOE: Collaborative Filtering for AutoML Model Selection https://arxiv.org/abs/1808.03233
1. Interestingly the foundations of this approach and MLP were invented / discovered around the same time about 66 years ago: 1957: https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Arnold_repr... 1958: https://en.wikipedia.org/wiki/Multilayer_perceptron 2. Another advantage of this approach is that it has only one class of parameters (the coefficients of the local activation functions) as opposed to MLP which has thr…
One might argue this via parsimony (Occam’s razor). Is this your thinking? / Anything else?
Earlier quoted context omitted.
In actual business processes there are lots of ML problems with fewer than 100 input dimensions. But for most of them decision trees are still competitive with neural networks or even outperform them.
Business processes don't need deep learning in the first place. It is just there because hype.
Lets call someone working at such a company Bob.
A restatement of your claim is that Bob decided to launch a model to live because of hype rather than because he could justify his promotion by pointing to the millions of dollars in increased revenue his switch produced. Bob of course did not make his decision based on hype. He made his decision because there were evaluation criteria in place for the launch. He was literally not allowed to launch things that didn't improve the system according to the evaluation criteria. As Bob didn't want to be fired for not doing anything at the company, he was forced to use a tool that worked to improve the evaluation according to the criteria that was specified. So he used the tool that worked. Hype might provide motivation to experiment, but it doesn't justify a launch.
I say this as someone whose literally seen transitions from decision trees to deep learning models on < 100 feature models which had multi-million dollar monthly revenue impacts.
Earlier quoted context omitted.
There's a ton actually. Just they tend to go through extra rounds of review (or never make it...) and never make it to HN unless there's special circumstances (this one is MIT and CIT). Unfortunately we've let PR become a very powerful force (it's always been a thing, but seems more influential now). We can fight against this by up voting things like this and if you're a reviewee, not focusing on sota (it's clearly b…
> never make it to HN unless there's special circumstances Yes, I agree. The two most common patterns I've noticed in research that does show up on HN are: 1) It outright improves, or has the potential to improve, applications currently used in production by many HN readers. In other words, it's not just navel-gazing. 2) The authors and/or their organizations are well-known, as you suggest.