Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gradients are not all you need

arxiv.org

Post reply on HN
Print view View original

41–50 of 71 posts

Re: Gradients are not all you need

#41

Post by V__ »

mumblemumble wrote: post #24
sho_hn wrote: post #9

Earlier quoted context omitted.

Remember the days after Rowhammer and Heartbleed, when every new security vulnerability needed its own catchy name and domain name website? This is the science version of that. Branding is eating the world.

It's inescapable. 40 years ago, when articles went into print publications, you'd just get your paper into a key print journal and then trust that everyone who gets it would at least look through the article headlines and read the abstracts of articles that seemed relevant to them. And it was manageable because you'd only have a few new issues rolling in per month. But arXiv had an average of 167 CS papers being subm…

Isn't this more a problem of not good enough curation? There are more papers now than ever, but the signal is probably getting worse and worse.

Re: Gradients are not all you need

#42

Post by smonn_ »

mumblemumble wrote: post #40
smonn_ wrote: post #29

Earlier quoted context omitted.

There's plenty of interesting neural network designs out there but they're being overshadowed by transformers due to their recent success. I personally thing that the main reason transformers work so well is because they actually step away from the multi layer perceptron stuff and introduce some structure and in a way sparsity.

Also, multi-head attention strikes me as being about as close to how language semantics seems to actually work in human brains as I've seen. Lots of caveats there, of course. First off, I don't know much about the neurology, I just have an amateur interest in second language acquisition research that sometimes brings me into contact with this sort of thing. On the ANN side, which is closer to my actual wheelhouse, we…

I'm currently working on a variation of a spiking neural network that learns by making and purging connections between neurons, which so far has been pretty interesting, though I am having a hard time getting it to output anything more than just the patterns it recognised. I did play around with adding its outputs to the input list, making it sort of recurrent but its practically impossible to decode anything thats going on inside of the network. Im thinking of tracking the inputs around to see what its doing right now, might be interesting to see it generate some sort of tree-like structure.

Re: Gradients are not all you need

#43

Post by neurobama »

smonn_ wrote: post #29
modeless wrote: post #17

Seems to me like the whole history of neural nets is basically crafting models with well-behaved gradients to make gradient descent work well. That, and models that can achieve high utilization of available hardware. The surprising thing is that models exist where the gradients are so well-behaved that we can learn GPT-4 level stuff.

There's plenty of interesting neural network designs out there but they're being overshadowed by transformers due to their recent success. I personally thing that the main reason transformers work so well is because they actually step away from the multi layer perceptron stuff and introduce some structure and in a way sparsity.

>There's plenty of interesting neural network designs out there

Where could a person learn more about these?

Re: Gradients are not all you need

#46

Post by kyllo »

smonn_ wrote: post #42
mumblemumble wrote: post #40

Earlier quoted context omitted.

Also, multi-head attention strikes me as being about as close to how language semantics seems to actually work in human brains as I've seen. Lots of caveats there, of course. First off, I don't know much about the neurology, I just have an amateur interest in second language acquisition research that sometimes brings me into contact with this sort of thing. On the ANN side, which is closer to my actual wheelhouse, we…

I'm currently working on a variation of a spiking neural network that learns by making and purging connections between neurons, which so far has been pretty interesting, though I am having a hard time getting it to output anything more than just the patterns it recognised. I did play around with adding its outputs to the input list, making it sort of recurrent but its practically impossible to decode anything thats g…

Are you familiar with the edge popup algorithm introduced in "What's Hidden in a Randomly Weighted Neural Network?" https://arxiv.org/abs/1911.13299v2

Seems relevant to what you're working on. It starts with a randomly initialized, overparameterized neural net, but instead of gradient descent backpropagation, it learns by deleting connection edges.

Re: Gradients are not all you need

#47

Post by nullc »

sho_hn wrote: post #9
unlikelymordant wrote: post #2

My one wish is that machine learning papers would use paper titles that actually described what the paper was about. I suppose there is a certain 'evolutionary pressure' where clever titles 'outcompete' dryer, more descriptive titles (or it seems that way). But i don't like it.

Remember the days after Rowhammer and Heartbleed, when every new security vulnerability needed its own catchy name and domain name website? This is the science version of that. Branding is eating the world.

> Remember the days after Rowhammer and Heartbleed, when every new security vulnerability needed its own catchy name and domain name website? This is the science version of that. Branding is eating the world.

Sorry, are you referring to brandpocalypse?

Re: Gradients are not all you need

#48

Post by mumblemumble »

V__ wrote: post #41
mumblemumble wrote: post #24

Earlier quoted context omitted.

It's inescapable. 40 years ago, when articles went into print publications, you'd just get your paper into a key print journal and then trust that everyone who gets it would at least look through the article headlines and read the abstracts of articles that seemed relevant to them. And it was manageable because you'd only have a few new issues rolling in per month. But arXiv had an average of 167 CS papers being subm…

Isn't this more a problem of not good enough curation? There are more papers now than ever, but the signal is probably getting worse and worse.

How does one, practically speaking, curate article publication in a world with arXiv?

Curation kind of requires gatekeepers, and good gatekeeping is a full-time job that needs to be paid for. In the days of print journals, there was both a motivation, because you can only print so many articles per month due to materials costs, and a steady revenue stream to pay for it, because everyone has to pay for the journals to get any access to articles at all. The Internet's been eroding both of those factors for a good quarter century now.

Re: Gradients are not all you need

#49

Post by miketery »

modeless wrote: post #17

Seems to me like the whole history of neural nets is basically crafting models with well-behaved gradients to make gradient descent work well. That, and models that can achieve high utilization of available hardware. The surprising thing is that models exist where the gradients are so well-behaved that we can learn GPT-4 level stuff.

> gradients are so well-behaved that we can learn GPT-4 level stuff

What are "well-behaved" gradients?

What type of GPT-4 level stuff?

Post reply on HN