Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-3: A Disappointing Paper?

greaterwrong.com

Post reply on HN
Print view View original

81–87 of 87 posts

Re: GPT-3: A Disappointing Paper?

#81

Post by p1esk »

6gvONxR4sf7o wrote: post #79
p1esk wrote: post #66

Earlier quoted context omitted.

This one uses FP16, so you just need to have a server with >350GB of RAM. 512GB of DDR4 would set you back around two grand. A total cost of a server for this would probably be under $5k. Comparable to a good gaming rig.

It uses FP16 yes, but the question was about average people running them on their PCs. I don't think most PCs have fp16 support, so you'd have to do it in fp32, doubling the size. It's likely not so fast on a CPU either with that size, especially when using FP32.

The ALUs (FPUs) in most CPUs are 64 bit (even more than that internally), but this does not matter, because we don't care how many bits our floats take inside the CPU, we care about how much space they take in our server's RAM. From our point of view, we supply weights and inputs to the CPU (both in FP16), CPU multiplies them (using 64 bit multipliers), and then spits out the result, which is cast to FP16, and that's what gets stored in memory.

Re: GPT-3: A Disappointing Paper?

#82

Post by phreeza »

The_rationalist wrote: post #68
phreeza wrote: post #60

Earlier quoted context omitted.

Not all models scale up well with more parameters. VAEs are not really exclusively a vision thing, they have been used in a variety of settings. Using VAEs for NLP is also nothing new, an early example is Bowman et al, 2015. https://arxiv.org/abs/1511.06349

What's new is that they are the first to eliminate the bottleneck that prevented VAES to beat transformers

What kind of bottleneck are you talking about? VAEs as such are really in a different category from transformers. They are primarily a tool to get to more structured latent spaces, which is not something transformers are good at in the first place.

Re: GPT-3: A Disappointing Paper?

#83

Post by clickok »

KKKKkkkk1 wrote: post #78
clickok wrote: post #53

Earlier quoted context omitted.

You have misunderstood Sutton's argument. Quoting Rich: One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The point isn't that improvements in our algorithms is unnecessary or unhelpful, rather that the algorithms we should focus on will be capable of…

I think this argument is neither here nor there. For computer vision problems, we use convnets, which are models inspired by a biological model of vision. By doing that we are embedding our preconceived notions of what vision is into our models instead of throwing compute and data at the problem. Earlier attempts using multi-layer perceptrons have been massive failures. Is this consistent with Sutton's analysis or co…

It's consistent with it.

Rich used to be very bullish on neural nets, then somewhat dismissive of them (due to the fragility/inadequacy of FCNs), and then increasingly enthusiastic as the renewed interest demonstrated that those problems could be overcome-- e.g., through better initialization, training, and (as you note) different architecture choices. His main concern was whether a method could keep working as more resources became available, as otherwise you would tautologically end up with something short of true artificial intelligence.

The important thing is that the technique can scale with increasing data or compute without hitting a hard or soft limit.

Re: GPT-3: A Disappointing Paper?

#84

Post by psb217 »

The_rationalist wrote: post #42
psb217 wrote: post #20

Earlier quoted context omitted.

FYI, the MELBO bound in that paper is invalid. Their perplexity numbers using the MELBO bound are also invalid.

Where is the error? How much would that change their score of 4.6?

The bound is completely invalid, as are the NLL/PPL numbers they report with the MELBO. Look at the equation. If they optimized it directly, it would be trivially driven to 0 by the identity function if we used a latent space equivalent to the input space. The MELBO just adds noiseless autoencoder reconstruction error to a constant offset equal to log of the test set size. This can be driven to zero by evaluating an average bound over test sets of size 1.

The mathematical/conceptual error is that they are assuming each test point is added to the "post-hoc aggregated" prior when they evaluate the bound. This is analogous to including a test point in the training set. Another version of this error would be adding a kernel centered on each test point to a kernel density estimator prior to evaluating test set NLL. In this case, obviously the best kernel has variance 0 and assigns arbitrarily high likelihood to the test data.

Re: GPT-3: A Disappointing Paper?

#86

Post by The_rationalist »

psb217 wrote: post #84
The_rationalist wrote: post #42

Earlier quoted context omitted.

Where is the error? How much would that change their score of 4.6?

The bound is completely invalid, as are the NLL/PPL numbers they report with the MELBO. Look at the equation. If they optimized it directly, it would be trivially driven to 0 by the identity function if we used a latent space equivalent to the input space. The MELBO just adds noiseless autoencoder reconstruction error to a constant offset equal to log of the test set size. This can be driven to zero by evaluating an…

Interesting, thx!

Re: GPT-3: A Disappointing Paper?

#87

Post by spion »

bondarchuk wrote: post #33
ideals wrote: post #30

Earlier quoted context omitted.

> As a human being... I know I'm cherry picking your post, sorry, but this line kinda stood out to me as funny but intriguing. Doesn't something like this go without saying? Or is GPT-3 advanced enough that we must now distinguish ourselves from the robots and the dogs?

Increase your context window ;) > But as an NLP academic researcher It is in opposition to that.

This hints at there being non-humans doing NLP research at the moment.
Post reply on HN