Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I don't trust papers out of “Top Labs” anymore

old.reddit.com

Post reply on HN
Print view View original

31–40 of 53 posts

Re: I don't trust papers out of “Top Labs” anymore

#31

Post by Veedrac »

mola wrote: post #21
Isinlor wrote: post #14

Eleuther.ai is just a bunch of random, but smart people without capital who decided on Twitter to recreate GPT-3. Recently they released GPT-NeoX-20B. They mainly coordinate on Discord. They got compute from some company for free. https://www.eleuther.ai/ Another group called BigScience got a grant from France to use a public institution supercomputer to train large language model in open. They are 71% done training…

How is this related? OP was complaining that most of these tons of compute papers don't really show mucjg advance theory wise. They say it's obvious by now that putting more compute would slightly push SOA. The comments there add that these fancy papers are hiding more important work by showing some pretty pictures and pumping the PR machines full power.

I'm sort of baffled whether people saying this sort of stuff actually read ML papers. Because this is just overtly not true, this idea that the majority of papers do scaling and nothing else. There are tons of papers exploring creative ideas, even the one mentioned in the critique, and even the subset of papers that are primarily about scale typically involve meaningful scientific discovery.

Re: I don't trust papers out of “Top Labs” anymore

#32

Post by Isinlor »

mola wrote: post #21
Isinlor wrote: post #14

Eleuther.ai is just a bunch of random, but smart people without capital who decided on Twitter to recreate GPT-3. Recently they released GPT-NeoX-20B. They mainly coordinate on Discord. They got compute from some company for free. https://www.eleuther.ai/ Another group called BigScience got a grant from France to use a public institution supercomputer to train large language model in open. They are 71% done training…

How is this related? OP was complaining that most of these tons of compute papers don't really show mucjg advance theory wise. They say it's obvious by now that putting more compute would slightly push SOA. The comments there add that these fancy papers are hiding more important work by showing some pretty pictures and pumping the PR machines full power.

I see there a rant that others have compute and he doesn't.

There is plenty of papers showing advance theory wise.

Some even show that big compute is necessary like "A Universal Law of Robustness via Isoperimetry":

> Classically, data interpolation with a parametrized model class is possible as long as the number of parameters is larger than the number of equations to be satisfied. A puzzling phenomenon in deep learning is that models are trained with many more parameters than what this classical theory would suggest. We propose a theoretical explanation for this phenomenon. We prove that for a broad class of data distributions and model classes, overparametrization is necessary if one wants to interpolate the data smoothly. Namely we show that smooth interpolation requires d times more parameters than mere interpolation, where d is the ambient data dimension. We prove this universal law of robustness for any smoothly parametrized function class with polynomial size weights, and any covariate distribution verifying isoperimetry. In the case of two-layers neural networks and Gaussian covariates, this law was conjectured in prior work by Bubeck, Li and Nagaraj. We also give an interpretation of our result as an improved generalization bound for model classes consisting of smooth functions.

https://arxiv.org/abs/2105.12806

I mean, you can be upset at the universe for the way it is, just what is the point?

Re: I don't trust papers out of “Top Labs” anymore

#33

Post by saiojd »

The real ugliness in jealousy comes from how it deceives the self. Consider the last part of this post:

"Is this really what we're comfortable with as a community? A handful of corporations and the occasional university waving their dicks at everyone because they've got the compute to burn and we don't"

I honestly think this kind of comment can only come from a place of jealousy. If someone is willing to spend a lot of money on an experiment, shouldn't you be glad it was done? A scientific field is not an athletic competition, where the rules are picked to measure your "worth as a competitor", and where the playing field has to be fair. The point is to move things forward. Many scientific fields have large, technical hurdles which require expensive equipment. If anything, computer science is a rare niche where it sometimes does not. If you want to build a career in a subfield where compute is important, you should do your best to get access to compute. If you are unable to do so while others are, then you might feel anger, shame, jealousy. But these feelings are really a problem you have with yourself, and not with the field of study.

Re: I don't trust papers out of “Top Labs” anymore

#34

Post by sdenton4 »

i_am_proteus wrote: post #18
phkahler wrote: post #6

It does seem like better algorithms to get similar results from smaller models should be prioritised. Rather than throwing more compute at a problem for 0.03 better score, show me one tenth the compute with a loss of 0.03 score. That would be impressive and far more useful.

While I am inclined to personally agree with your sentiment, I don't think I have better insights than Richard Sutton: http://incompleteideas.net/IncIdeas/BitterLesson.html "The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin."

There's a great motivation for small model work for big-model results: More efficient use of compute can be leveraged to make big models effectively bigger. Small-model architectural innovations are computational leverage. You can even see the convolution operation in this light; it's much more efficient than the 'giant dense matrix' approach.

EfficientNet is an exemplar of this approach; they made much better small models, and wound up with much higher quality big models as a result of having better architecture overall: https://arxiv.org/pdf/1905.11946.pdf

We're currently seeing some great results with more efficient attention layers, which will make the current 'big' models much more efficient... And unlock a next generation of higher quality big models.

Re: I don't trust papers out of “Top Labs” anymore

#36

Post by ezoe »

Veedrac wrote: post #15
togaen wrote: post #2

"Jeff Dean spent enough money to feed a family of four for half a decade to get a 0.03% improvement on CIFAR-10." Nailed it.

That quote is disingenuous. Do people really think that... * Jeff Dean, lead of Google's AI division, wrote a paper with all that complexity to get SOTA on CIFAR-10? * Jeff Dean, whose salary is sometimes estimated as $3m/y and is responsible for the direction of research of many more, is unreasonable for using * going from a 0.6% error rate to a 0.57% error rate is reasonably summarized as ‘a 0.03% improvement’, ign…

Still, it's 0.03% difference, or 3 images difference out of 10k images in CIFAR-10. Just 3 images.

Re-training SotA with a different random seed may make its score 0.03% difference. Or there was a wrong calculation in 17,810 TPU core-hours due to faulty hardware or cosmic ray hit which cause the final produce model 0.03% difference.

Re: I don't trust papers out of “Top Labs” anymore

#37

Post by Der_Einzige »

Strilanc wrote: post #19

They explicitly say they trust the results. They're complaining that top labs use lots of compute, so the results aren't relevant to someone who can't. They give an example where a paper used 18K TPU core hours. It's easy to find papers that use millions of core hours. IMO, asking AI people to not use expensive compute is like asking astronomers to please stop using expensive telescopes. The opposite side of this arg…

The main problem is that it kills the double blind nature of peer review.

If your paper says you trained on 1000TPUs for weeks, we all know you work at Google brain.

This is subversive for our field. It's really really bad that these authors are virtually guaranteed to be accepted for these reasons alone.

Re: I don't trust papers out of “Top Labs” anymore

#38

Post by Der_Einzige »

mjburgess wrote: post #22

A modern "AI" models have c. 200bn parameters, say. At 32bit/param that's c. 6TB. At 6 bytes/word, 1T words, or more words than are in all books that have ever been written. NNs, and models of this kind, are just search engines. They store a compression of of everything ever written, and prediction is just googling through it. Models performance exponential in parameter count should be just ignored by research. This…

The deep connections between compression and prediction are not always obvious to those not in the field.

To illustrate just how much they are the same, here is (at one point SOTA) lossless text compression with GPT-2

https://bellard.org/libnc/gpt2tc.html

Re: I don't trust papers out of “Top Labs” anymore

#39

Post by mjburgess »

Der_Einzige wrote: post #38
mjburgess wrote: post #22

A modern "AI" models have c. 200bn parameters, say. At 32bit/param that's c. 6TB. At 6 bytes/word, 1T words, or more words than are in all books that have ever been written. NNs, and models of this kind, are just search engines. They store a compression of of everything ever written, and prediction is just googling through it. Models performance exponential in parameter count should be just ignored by research. This…

The deep connections between compression and prediction are not always obvious to those not in the field. To illustrate just how much they are the same, here is (at one point SOTA) lossless text compression with GPT-2 https://bellard.org/libnc/gpt2tc.html

Well.. I think they're over-stated because of the current wave of AI basically only having naïve compression as its tool.

Is the concept `addition` a compression of the space `(Int, Int, Int)` ?

If you want to say it is, OK for some definition of compression. But that compression isnt "mere" in the modern AI sense, it's "exponentially dense".

In that my concept `addition` can generate arbitrarily large amounts of that decompressed space, which is infinite in size.

There's a kind of trick played in the marketing here: since NNs compress, and since learning "can be seen as compression", NNs learn... no, because NNs aren't "exponentially dense", they're "exponentially large" -- I'd claim, the opposite of learning!

Re: I don't trust papers out of “Top Labs” anymore

#40

Post by Veedrac »

ezoe wrote: post #36
Veedrac wrote: post #15

Earlier quoted context omitted.

That quote is disingenuous. Do people really think that... * Jeff Dean, lead of Google's AI division, wrote a paper with all that complexity to get SOTA on CIFAR-10? * Jeff Dean, whose salary is sometimes estimated as $3m/y and is responsible for the direction of research of many more, is unreasonable for using * going from a 0.6% error rate to a 0.57% error rate is reasonably summarized as ‘a 0.03% improvement’, ign…

Still, it's 0.03% difference, or 3 images difference out of 10k images in CIFAR-10. Just 3 images. Re-training SotA with a different random seed may make its score 0.03% difference. Or there was a wrong calculation in 17,810 TPU core-hours due to faulty hardware or cosmic ray hit which cause the final produce model 0.03% difference.

The problem with this sort of argument against caring about SOTA scores is that there is only so much luck to go around. While any individual 5% reduction in error rates could theoretically be highly influenced by luck, if you have a chain of small reductions in error rates, such that the difference between the first and the last is more like a factor of 2, then you know that somewhere in the middle of that, even if any individual improvement is suspect, there must have been real, gradual improvement.

It isn't that important on CIFAR-10 any more, which is pretty much a solved benchmark, but CIFAR was only solved because of such incremental progress, and papers focusing on moving the state of the art use newer, much harder benchmarks.

Post reply on HN