Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I don't trust papers out of “Top Labs” anymore

old.reddit.com

Post reply on HN
Print view View original

21–30 of 53 posts

Re: I don't trust papers out of “Top Labs” anymore

#21

Post by mola »

Isinlor wrote: post #14

Eleuther.ai is just a bunch of random, but smart people without capital who decided on Twitter to recreate GPT-3. Recently they released GPT-NeoX-20B. They mainly coordinate on Discord. They got compute from some company for free. https://www.eleuther.ai/ Another group called BigScience got a grant from France to use a public institution supercomputer to train large language model in open. They are 71% done training…

How is this related? OP was complaining that most of these tons of compute papers don't really show mucjg advance theory wise. They say it's obvious by now that putting more compute would slightly push SOA. The comments there add that these fancy papers are hiding more important work by showing some pretty pictures and pumping the PR machines full power.

Re: I don't trust papers out of “Top Labs” anymore

#22

Post by mjburgess »

A modern "AI" models have c. 200bn parameters, say. At 32bit/param that's c. 6TB. At 6 bytes/word, 1T words, or more words than are in all books that have ever been written.

NNs, and models of this kind, are just search engines. They store a compression of of everything ever written, and prediction is just googling through it.

Models performance exponential in parameter count should be just ignored by research. This category of performance is already established by research, more compute and more historical data stored, isnt an interesting research result.

Re: I don't trust papers out of “Top Labs” anymore

#23

Post by j7ake »

It’s even worse in biology where some labs consistently publish in Nature, Science, Cell. Some of the papers are outright fraudulent. Don’t even trust the numbers.

At least for ML you can mostly reproduce the results, even in if they’re not that interesting.

Re: I don't trust papers out of “Top Labs” anymore

#25

Post by plorkyeran »

Strilanc wrote: post #19

They explicitly say they trust the results. They're complaining that top labs use lots of compute, so the results aren't relevant to someone who can't. They give an example where a paper used 18K TPU core hours. It's easy to find papers that use millions of core hours. IMO, asking AI people to not use expensive compute is like asking astronomers to please stop using expensive telescopes. The opposite side of this arg…

We already know that more compute hours give better results, and a paper which simply consists of rerunning previous work but with 100x the compute hours for .03% better results has not discovered anything new, and there's no point in reading that paper.

Re: I don't trust papers out of “Top Labs” anymore

#26

Post by ta988 »

You shouldn't "trust" papers, stay critical and verify. Wherever they come from. There is a lot of politics, grad students eager to graduate so they cut corners, cheating PIs, cheatings statisticians... (I've witnessed each of these during my career). What you should trust is when things get built upon other works (from other groups) or when it simply gets reproduced. This does not eliminate the risk of fraud or error but greatly reduce it. The same way do not trust claims from companies based on a single paper especially if the company is run by one of the authors. Again it is just my limited experience but most of the ones I have seen were just full of overblown claims and they just hoped they could jump the ship before it got discovered.

Re: I don't trust papers out of “Top Labs” anymore

#27

Post by ezoe »

CIFAR-10 is consists of 10,000 test images. So 0.03% of CIFAR-10 is 3 images.

At this tiny number, the randomness is starting to affect the scores. Like labeling mistake of test data by human. Maybe, training SotA with different random seeds make its score 0.03% better or worse.

Hell, 17,810 TPU core-hours is a huge number. You can't ignore the work of randomness. What if a cosmic ray hit a specific memory cell which cause the soft memory error, causing a single wrong calculation which ultimately cause the final trained model 0.03% difference?

So, it's more like: "Jeff Dean spent enough money to feed a family of four for half a decade to get a 0.03% of winning lottery on CIFAR-10."

Re: I don't trust papers out of “Top Labs” anymore

#28

Post by iakh »

plorkyeran wrote: post #25
Strilanc wrote: post #19

They explicitly say they trust the results. They're complaining that top labs use lots of compute, so the results aren't relevant to someone who can't. They give an example where a paper used 18K TPU core hours. It's easy to find papers that use millions of core hours. IMO, asking AI people to not use expensive compute is like asking astronomers to please stop using expensive telescopes. The opposite side of this arg…

We already know that more compute hours give better results, and a paper which simply consists of rerunning previous work but with 100x the compute hours for .03% better results has not discovered anything new, and there's no point in reading that paper.

I'm not in this space, but I'd expect that they wouldn't know the results beforehand and just publishing the results even when not a major improvement tells the community that they don't need to spend the extra 99x compute. That seems valuable to some degree. Or is the argument that there wasn't any improvement to be had so why even test the 100x extra?

Re: I don't trust papers out of “Top Labs” anymore

#29

Post by Veedrac »

plorkyeran wrote: post #25
Strilanc wrote: post #19

They explicitly say they trust the results. They're complaining that top labs use lots of compute, so the results aren't relevant to someone who can't. They give an example where a paper used 18K TPU core hours. It's easy to find papers that use millions of core hours. IMO, asking AI people to not use expensive compute is like asking astronomers to please stop using expensive telescopes. The opposite side of this arg…

We already know that more compute hours give better results, and a paper which simply consists of rerunning previous work but with 100x the compute hours for .03% better results has not discovered anything new, and there's no point in reading that paper.

That might be a reasonable criticism if it remotely reflected reality, but scaling has repeatedly shown to produce qualitatively stronger models, by large margins, doing things that would seem unimaginable for smaller models.

Re: I don't trust papers out of “Top Labs” anymore

#30

Post by whiplash451 »

plorkyeran wrote: post #25
Strilanc wrote: post #19

They explicitly say they trust the results. They're complaining that top labs use lots of compute, so the results aren't relevant to someone who can't. They give an example where a paper used 18K TPU core hours. It's easy to find papers that use millions of core hours. IMO, asking AI people to not use expensive compute is like asking astronomers to please stop using expensive telescopes. The opposite side of this arg…

We already know that more compute hours give better results, and a paper which simply consists of rerunning previous work but with 100x the compute hours for .03% better results has not discovered anything new, and there's no point in reading that paper.

We are not at this point yet. Recent work shows that more compute and more data let Transformers beat convnets on computer vision tasks. This is a lot more insightful than « more compute gets you a little further ».
Post reply on HN