GLMs in turn generalize logistic-, linear and other popular regression models.
Neural GAMs with learned basis functions have already been proposed, so I'm a bit surprised that the prior art is not mentioned in this new paper. Previous applications focused more on interpretability.
I was under the impression that graph neural nets already trained learnable functions on graph edges rather than nodes, albeit typically on a fully connected graph. Is there any comparison to just a basic GNN here?
This makes me wonder what you could achieve if instead of iteratively growing the grid, or worrying about pruning or regularization, you governed network topology with some sort of evolutionary algorithm.
You can do much better by growing an AST with memoization and non-linear regression. So much so, the EVO folks gave a best paper to a non-EVO, deterministic algorithm at their conference https://seminars.math.binghamton.edu/ComboSem/worm-chiu.pge_... (author)
Yes seconding this. If you want a broad view of ML IMHO the best places to look at are conference proceedings. The typical review process is imperfect so that still doesn't show you all the interesting work out there (which you mention), but it is still a start wrt diversity of research. I follow LLMs closely but then going through proceedings means I come across exciting research like these [1],[2],[3]. References:…
Honestly, these days I just rely on arxiv. The conferences are so noisy that it is hard to really tell what's useful and what's crap. Twitter is a bit better but still a crap shoot. So as far as it seems to me, there's no real good signal to use to differentiate. And what's the point of journals/conferences if not to provide some reasonable signal? If it is a slot machine, it is useless. And I feel like we're far too…
I agree with almost all you said except that Twitter is better than top conferences, and I take a contrarian view that reviewers slow down AGI with requests for additional experiments. Without going into specifics, which you can probably guess based on your background, too many ideas that work well, even optimally, at small scale fail horribly at large scale. Other ideas that work at super specialized settings don’t transfer or don’t generalize. The saving of two or three dimensions for exact symmetry operations is super important when you deal with handful of dimensions and is often irrelevant or slowing down training a lot when you already deal with tens of thousands of dimensions. Correlations in huge multimodal datasets are way more complicated than most humans can grasp and we will not get to AGI before we can have a large enough group of people dealing with such data routinely. It is very likely detrimental for our progress to AGI that we lack abundant hardware for academics and hobbyists to contribute frontier experiments, however we don’t do anybody a favor by increasing the entropy of the publications in the huge ML conferences. This particular work listed in HN stands out despite lack of scaling and will probably make it in a top conference (perhaps with some additional background citations) but not everything that is merely novel should simply make it to ICLR or neurIPS or ICML, otherwise we could have a million papers in each in a few years from today and nobody would be the wiser.
I had a European peasant in the 1600-1700s in mind when I wrote about the amount of work. During the season, they worked all day; off-season, they had "free time" that went into taking care of the household, inventory, etc., so it's still work. Can't quickly find a reliable source in English I could link, so I can be wrong here. "Better" was referring to what OP wrote in the top comment. I guess 10x faster, 10x longe…
Sorry, I can't fit that with what you wrote earlier: "12 hrs/7 days per week to not die from hunger". Those peasants payed taxes, i.e. some of their work was exploited by an army or a priest rather than hunger, and as you mention, they did not work "12 hrs/7 days per week". Do you have a better example?
Yes arxiv is a good first source too. I mentioned conferences as a way to get exposed to diversity, but not necessarily (sadly) merit. It has been my experience as an author and reviewer both that review quality has plummeted over the years for the most part. As a reviewer I had to struggle with the ills of "commission and omission" both, i.e., (a) convince other reviewers to see an idea (from a trendy area such as i…
I've stopped considering novelty at all. The only thing I now consider is if the precise technique has been done before. If not, well I've seen pretty small things change results dramatically. The pattern I've seen that scares me more is that when authors do find simple but effective changes, they end up convoluting the ideas because simplicity and clarity is often confused with novelty. And honestly, revisiting idea…
Sorry to hear all this (after writing my other sibling comment). Please don’t lose faith in the review process. It is still useful. Until the AGI can be better reviewers, which is hopefully not too far in the future.
Sorry, I can't fit that with what you wrote earlier: "12 hrs/7 days per week to not die from hunger". Those peasants payed taxes, i.e. some of their work was exploited by an army or a priest rather than hunger, and as you mention, they did not work "12 hrs/7 days per week". Do you have a better example?
This entire line of argument is just pointless.
You probably placed this wrong? I'm not driving a line of argument here.
Yes arxiv is a good first source too. I mentioned conferences as a way to get exposed to diversity, but not necessarily (sadly) merit. It has been my experience as an author and reviewer both that review quality has plummeted over the years for the most part. As a reviewer I had to struggle with the ills of "commission and omission" both, i.e., (a) convince other reviewers to see an idea (from a trendy area such as i…
I've stopped considering novelty at all. The only thing I now consider is if the precise technique has been done before. If not, well I've seen pretty small things change results dramatically. The pattern I've seen that scares me more is that when authors do find simple but effective changes, they end up convoluting the ideas because simplicity and clarity is often confused with novelty. And honestly, revisiting idea…
Sorry to hear that. My experiences haven't been very different. I really can't tell if the current review process is the least bad among alternatives or is there something better (if so, what is it?).
Very unfriendly. The symbolic library (type of activations) requires a branching at the very core of the kernel. GPU will need to serialized on these operations warp-wise. To optimize, you might want to do a scan operation beforehand and dispatch to activation funcs in a warp specialized way, this, however, makes the global memory read/write non-coalesced. You then may sort the input based on type of activations and…
Wouldn't it be faster to calculate every function type and then just multiply them by 0s or 1s to keep the active ones?