Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gradients are not all you need

arxiv.org

Post reply on HN
Print view View original

11–20 of 71 posts

Re: Gradients are not all you need

#11

Post by dauertewigkeit »

unlikelymordant wrote: post #2

My one wish is that machine learning papers would use paper titles that actually described what the paper was about. I suppose there is a certain 'evolutionary pressure' where clever titles 'outcompete' dryer, more descriptive titles (or it seems that way). But i don't like it.

At some point you run into the problem where titles become useless because there are 100 papers on the same exact topic with very slight variation in the title. At this point people use surnames and date to cite papers.

But then the title can become something catchy that will give you more visibility.

Re: Gradients are not all you need

#12

Post by preommr »

sho_hn wrote: post #9
unlikelymordant wrote: post #2

My one wish is that machine learning papers would use paper titles that actually described what the paper was about. I suppose there is a certain 'evolutionary pressure' where clever titles 'outcompete' dryer, more descriptive titles (or it seems that way). But i don't like it.

Remember the days after Rowhammer and Heartbleed, when every new security vulnerability needed its own catchy name and domain name website? This is the science version of that. Branding is eating the world.

> Branding is eating the world.

Not realizing that it always has is why it's still effective.

History is written by the victors and victors tend to have the best marketting.

Re: Gradients are not all you need

#13

Post by [deleted] »

preommr wrote: post #12
sho_hn wrote: post #9

Earlier quoted context omitted.

Remember the days after Rowhammer and Heartbleed, when every new security vulnerability needed its own catchy name and domain name website? This is the science version of that. Branding is eating the world.

> Branding is eating the world. Not realizing that it always has is why it's still effective. History is written by the victors and victors tend to have the best marketting.

[deleted]

Re: Gradients are not all you need

#15

Post by cubefox »

satvikpendem wrote: post #8
unlikelymordant wrote: post #2

My one wish is that machine learning papers would use paper titles that actually described what the paper was about. I suppose there is a certain 'evolutionary pressure' where clever titles 'outcompete' dryer, more descriptive titles (or it seems that way). But i don't like it.

The clever titles are more brandable. See the citations for "Attention is all you need" or "Chinchilla limit" versus more mundane titles.

The Chinchilla paper was called "Training Compute-Optimal Large Language Models", which is exactly on point.

Re: Gradients are not all you need

#16

Post by dragonwriter »

cubefox wrote: post #15
satvikpendem wrote: post #8

Earlier quoted context omitted.

The clever titles are more brandable. See the citations for "Attention is all you need" or "Chinchilla limit" versus more mundane titles.

The Chinchilla paper was called "Training Compute-Optimal Large Language Models", which is exactly on point.

I mean, it would be slightly more accurate if it had been “Compute-Optimal Training of Large Language Models”, since the models so-trained aren’t the thing that is compute-optimal, the training is compute-optimal.

But… yeah, its hardly a title chosen for marketing rather than description.

Re: Gradients are not all you need

#17

Post by modeless »

Seems to me like the whole history of neural nets is basically crafting models with well-behaved gradients to make gradient descent work well. That, and models that can achieve high utilization of available hardware. The surprising thing is that models exist where the gradients are so well-behaved that we can learn GPT-4 level stuff.

Re: Gradients are not all you need

#18

Post by skybrian »

optimalsolver wrote: post #3
unlikelymordant wrote: post #2

My one wish is that machine learning papers would use paper titles that actually described what the paper was about. I suppose there is a certain 'evolutionary pressure' where clever titles 'outcompete' dryer, more descriptive titles (or it seems that way). But i don't like it.

I'd like to imagine quirky titles make them harder for other researchers to find, leading to lower citation counts, but that may be wishful thinking.

I don’t think it makes much difference. If anything, it might help? It’s easier to search for a paper if you remember its name. If not, you can search on author or words in the abstract.

The problem isn’t quirky titles, it’s with websites like Hacker News that only display a headline and not the abstract.

Re: Gradients are not all you need

#19

Post by sroussey »

dragonwriter wrote: post #16
cubefox wrote: post #15

Earlier quoted context omitted.

The Chinchilla paper was called "Training Compute-Optimal Large Language Models", which is exactly on point.

I mean, it would be slightly more accurate if it had been “Compute-Optimal Training of Large Language Models”, since the models so-trained aren’t the thing that is compute-optimal, the training is compute-optimal. But… yeah, its hardly a title chosen for marketing rather than description.

But we all know what you mean when you say “the Chinchilla paper”

Re: Gradients are not all you need

#20

Post by moritzwarhier »

dragonwriter wrote: post #16
cubefox wrote: post #15

Earlier quoted context omitted.

The Chinchilla paper was called "Training Compute-Optimal Large Language Models", which is exactly on point.

I mean, it would be slightly more accurate if it had been “Compute-Optimal Training of Large Language Models”, since the models so-trained aren’t the thing that is compute-optimal, the training is compute-optimal. But… yeah, its hardly a title chosen for marketing rather than description.

Stupid question, is training considered a part of the model? Or is this only common parlance for GPT (the P).
Post reply on HN