Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-3 is no longer the only game in town

lastweekin.ai

Post reply on HN
Print view View original

131–140 of 217 posts

Re: GPT-3 is no longer the only game in town

#131

Post by 6gvONxR4sf7o »

sillysaurusx wrote: post #128
warning26 wrote: post #3

Neat to see more models getting closer, thought it appears only one so far has exceeded GPT-3's 175B parameters. That said, what I'm really curious is how those other models stack up against GPT-3 in terms of performance -- does anyone know of any comparisons?

I’m surprised that no one has answered for three hours! The answer is at https://github.com/kingoflolz/mesh-transformer-jax It has detailed comparisons and a full breakdown of the performance, courtesy of Eleuther.

I was so frustrated when that was first announced because it didn't include those metrics, and everyone ate it up like the models were equivalent.

Re: GPT-3 is no longer the only game in town

#132

Post by 6gvONxR4sf7o »

Nextgrid wrote: post #102
minimaxir wrote: post #92

Earlier quoted context omitted.

Fortunately, costs for training superlarge models are coming down rapidly thanks to TPUs (which was the approach used to train GPT-J 6B) and DeepSpeed improvements.

Are there any TPUs that can be purchased off-the-shelf and then owned, like you can do with a CPU or GPU? Or are you just limited to paying rent to cloud providers and ultimately being at their mercy when it comes to pricing, ToS, etc?

No, but you probably aren't going to buy an A100 either, so it's a moot point.

Re: GPT-3 is no longer the only game in town

#133

Post by lostdog »

CodeGlitch wrote: post #112
lostdog wrote: post #91

Earlier quoted context omitted.

The difference between ML and symbolic AI is that ML works and symbolic AI doesn't. At my job, dropping the computational load of our ML models is heavily invested in, and every success is celebrated. Everybody wants it to be easier and cheaper to train high quality models, but some things are still intrinsically hard.

> The difference between ML and symbolic AI is that ML works and symbolic AI doesn't. IBM managed to beat Garry Kasperov using symbolic AI did they not? So in what way does it not work?

> IBM managed to beat Garry Kasperov using symbolic AI did they not? So in what way does it not work?

Ok, I should be clearer. ML approaches are way way better than symbolic approaches. Given almost any problem, it is much much easier to make an ML approach work than any symbolic approach.

Yes, chess was first solved symbolically, but it's since been solved by ML better and more easily, to the point that stockfish now incorporates neural nets [1]. ML has also given extremely high levels of performance on Go, Starcraft, DoTA, and on protein folding, image recognition, text processing, speech recognition, and pretty much everything else.

I would challenge you to name any (non-simple) problem where traditional AI methods are still state of the art.

[1] https://stockfishchess.org/blog/2020/stockfish-12/

Re: GPT-3 is no longer the only game in town

#134

Post by [deleted] »

iamcurious wrote: post #57
ComodoHacker wrote: post #51

Are we heading to the (distant) future where to make progress in any field you have to spend big $$$ to train a model?

We are already there. Machine learning is the flavor of A.I. that keeps business barriers of entry high. If we had invested in symbolic A.I., things would be different. A similar thing happens with programming language flavors. PHP lowers barriers of entry so it is discredited by the incumbents.

[deleted]

Re: GPT-3 is no longer the only game in town

#136

Post by YetAnotherNick »

ComodoHacker wrote: post #51

Are we heading to the (distant) future where to make progress in any field you have to spend big $$$ to train a model?

Training model is getting cheaper. GPT-3 is one of the very few countable examples where it is so expensive. In the end it all depends on the size of data you have that you could scale up the model without overfitting. And the internet text data is one of the only data size is this big.

Re: GPT-3 is no longer the only game in town

#137

Post by supperburg »

Simon321 wrote: post #86
supperburg wrote: post #52

I have supported my beliefs on this topic in these threads to the point of exhausting myself. The tools that we use to find these agents are the underpinning of AGI, it’s coming way faster than even most people here appreciate, this development is intrinsically against the interest of human beings. Please stop and think, please.

I argue it's very much in the interest of human beings. It has been since we first picked up a rock and used it has a hammer. It's the ultimate tool and has the potential to bring unseen prosperity.

It won’t. You’re wrong.this is the perfect illustration. You think a rock is good therefore AI is good. You’re just unbelievably wrong.

Re: GPT-3 is no longer the only game in town

#138

Post by CodeGlitch »

lostdog wrote: post #133
CodeGlitch wrote: post #112

Earlier quoted context omitted.

> The difference between ML and symbolic AI is that ML works and symbolic AI doesn't. IBM managed to beat Garry Kasperov using symbolic AI did they not? So in what way does it not work?

> IBM managed to beat Garry Kasperov using symbolic AI did they not? So in what way does it not work? Ok, I should be clearer. ML approaches are way way better than symbolic approaches. Given almost any problem, it is much much easier to make an ML approach work than any symbolic approach. Yes, chess was first solved symbolically, but it's since been solved by ML better and more easily, to the point that stockfish no…

Thanks for clearing that up, I do agree that ML-based AI has surpassed symbolic approaches in every field.

Re: GPT-3 is no longer the only game in town

#139

Post by CodeGlitch »

PeterisP wrote: post #127
CodeGlitch wrote: post #112

Earlier quoted context omitted.

> The difference between ML and symbolic AI is that ML works and symbolic AI doesn't. IBM managed to beat Garry Kasperov using symbolic AI did they not? So in what way does it not work?

1. There's a world of problems (such as "perception-related" e.g. vision and NLP) which we tried to solve for decades with symbolic AI and got worse results than what nowadays first-year students can do as a homework with ML; 2. For your example of chess, for some time now ML engines are pretty much untouchable by engines based on pre-ML methods.

Yes I agree with all your points - I was however responding to the point being made that symbolic AI "wasn't useful"...which in the past it was. Perhaps in the future some new method or breakthrough will mean it becomes useful once again?

Re: GPT-3 is no longer the only game in town

#140

Post by worldsayshi »

trothamel wrote: post #55
worldsayshi wrote: post #41

Earlier quoted context omitted.

But what about the potential for intellectual property problems?

That's beside the point, which is that the output copilot produces is useful.

I don't see how that's besides the point. How can it be that useful if the output is a such legal mystery?

I'd love to use it but not when there's such a risk of compromising the code base.

Post reply on HN