Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-3 is no longer the only game in town

lastweekin.ai

Post reply on HN
Print view View original

151–160 of 217 posts

Re: GPT-3 is no longer the only game in town

#151

Post by benatkin »

renewiltord wrote: post #50
sva_ wrote: post #34

Earlier quoted context omitted.

> Interviewer: Are you in favor of a carbon tax? > Mr. Powell: I don’t want to get into the details of taxes. > Interviewer: Are you in favor of a cap and trade system? > Mr. Powell: I don’t want to get into the details of a cap and trade system. > Interviewer: How do you think a global pandemic would impact the financial system? > Mr. Powell: A pandemic will have impact on the economy. It will have an impact on the…

I, too, that sounded like Eliza. Anyway, it looks like that’s a small excerpt from the conversation.

It looks like the dialogue is only on the human end. The chatbot is treating each question as the first. I think it sounds a lot like Biden. I prefer that to Trump, but don't like either sort of conversation!

Re: GPT-3 is no longer the only game in town

#152

Post by _game_of_life »

goodside wrote: post #144
lostdog wrote: post #133

Earlier quoted context omitted.

> IBM managed to beat Garry Kasperov using symbolic AI did they not? So in what way does it not work? Ok, I should be clearer. ML approaches are way way better than symbolic approaches. Given almost any problem, it is much much easier to make an ML approach work than any symbolic approach. Yes, chess was first solved symbolically, but it's since been solved by ML better and more easily, to the point that stockfish no…

“I would challenge you to name any (non-simple) problem where traditional AI methods are still state of the art.” Lossless file compression. As far as I know none of the algorithms in widespread use are neural-based, despite the fact that compression is clearly a rich statistical modeling problem, at least on par with GPT-3-style language understanding in difficulty. There are published attempts to solve the problem…

I'm far from an expert in this subject but doesn't this ranking of large text compression algorithms with NNCP coming first suggest that neural-nets are pretty great at compression?

http://mattmahoney.net/dc/text.html

https://bellard.org/nncp/

I don't see examples of high performing symbolic AI based compression algorithms anywhere, but again I am very ignorant, do you have examples?

Re: GPT-3 is no longer the only game in town

#153

Post by zebraflask »

rg111 wrote: post #93

The future is not as dark as it seems because of the rat race of megacorps. You can use reduced versions of language models with extremely good results. I was involved in training the first-ever GPT2 for Bengali language, but with 117 million parameters. It took a month's effort (training + writing code + setup) and about $6k in TPU cost, but Google Cloud covered it. Anyway, it is surprisingly good. We fine-tuned the…

That is a fantastic result - nagging question - these work best on predictable things. How much of Bengali poetry is predictable?

Re: GPT-3 is no longer the only game in town

#154

Post by YeGoblynQueenne »

lostdog wrote: post #133
CodeGlitch wrote: post #112

Earlier quoted context omitted.

> The difference between ML and symbolic AI is that ML works and symbolic AI doesn't. IBM managed to beat Garry Kasperov using symbolic AI did they not? So in what way does it not work?

> IBM managed to beat Garry Kasperov using symbolic AI did they not? So in what way does it not work? Ok, I should be clearer. ML approaches are way way better than symbolic approaches. Given almost any problem, it is much much easier to make an ML approach work than any symbolic approach. Yes, chess was first solved symbolically, but it's since been solved by ML better and more easily, to the point that stockfish no…

>> I would challenge you to name any (non-simple) problem where traditional AI methods are still state of the art.

Theorem proving, classical planning, SAT solving, robotics, search, in particular adversarial search, program induction, knowledge representation.

Plus all the stuff that used to be considered "AI" but aren't anymore, like rule-based systems (e.g. for fraud detection) etc.

Sorry, I know you asked for only one.

Re: GPT-3 is no longer the only game in town

#155

Post by avmich »

lostdog wrote: post #133
CodeGlitch wrote: post #112

Earlier quoted context omitted.

> The difference between ML and symbolic AI is that ML works and symbolic AI doesn't. IBM managed to beat Garry Kasperov using symbolic AI did they not? So in what way does it not work?

> IBM managed to beat Garry Kasperov using symbolic AI did they not? So in what way does it not work? Ok, I should be clearer. ML approaches are way way better than symbolic approaches. Given almost any problem, it is much much easier to make an ML approach work than any symbolic approach. Yes, chess was first solved symbolically, but it's since been solved by ML better and more easily, to the point that stockfish no…

> I would challenge you to name any (non-simple) problem where traditional AI methods are still state of the art.

Did ML methods best classic AI in dialog comprehension, say to the level of SHRDLU? I'm curious, can ML system do that - https://en.wikipedia.org/wiki/SHRDLU ?

Re: GPT-3 is no longer the only game in town

#156

Post by Dylan16807 »

supermatt wrote: post #56
qwertox wrote: post #47

If I were to run GPT-3 on my 70000 browser bookmarks, what kind of insights could I get from that? Only by analyzing the page title (from the bookmark, not by re-fetching the url) and eventually also the domain name.

GPT-3 is a text generator, so i doubt you would get anything of use. You cant even supply such a large input to GPT-3.

It's pretty good at extending lists. That might require some sorting first.

Re: GPT-3 is no longer the only game in town

#157

Post by YeGoblynQueenne »

PeterisP wrote: post #127
CodeGlitch wrote: post #112

Earlier quoted context omitted.

> The difference between ML and symbolic AI is that ML works and symbolic AI doesn't. IBM managed to beat Garry Kasperov using symbolic AI did they not? So in what way does it not work?

1. There's a world of problems (such as "perception-related" e.g. vision and NLP) which we tried to solve for decades with symbolic AI and got worse results than what nowadays first-year students can do as a homework with ML; 2. For your example of chess, for some time now ML engines are pretty much untouchable by engines based on pre-ML methods.

>> 1. There's a world of problems (such as "perception-related" e.g. vision and NLP) which we tried to solve for decades with symbolic AI and got worse results than what nowadays first-year students can do as a homework with ML;

Perception tasks were traditionally attempted with statistical machine learning approaches rather than symbolic AI, for example the Perceptron was a very early neural network that was used in machine vision, created by Frank Rosenblatt in 1958.

A lot of that research was carried out under the rubrik of "pattern recognition" rather than machine learning. In any case, no, "we" did not try "to solve [those problms] for decades with symbolic AI". Symbolic AI has traditionally focused on reasoning, which is generally considered to be on some kind of separate level to perception.

As to chess engines, they're still symbolic-statistical hybrids. E.g. the Alpha-x family combines Monte Carlo Tree Search with neural nets that learn an evaluation function etc.

Re: GPT-3 is no longer the only game in town

#158

Post by timschmidt »

Nextgrid wrote: post #102
minimaxir wrote: post #92

Earlier quoted context omitted.

Fortunately, costs for training superlarge models are coming down rapidly thanks to TPUs (which was the approach used to train GPT-J 6B) and DeepSpeed improvements.

Are there any TPUs that can be purchased off-the-shelf and then owned, like you can do with a CPU or GPU? Or are you just limited to paying rent to cloud providers and ultimately being at their mercy when it comes to pricing, ToS, etc?

https://aiyprojects.withgoogle.com/edge-tpu/

Re: GPT-3 is no longer the only game in town

#159

Post by YeGoblynQueenne »

avmich wrote: post #155
lostdog wrote: post #133

Earlier quoted context omitted.

> IBM managed to beat Garry Kasperov using symbolic AI did they not? So in what way does it not work? Ok, I should be clearer. ML approaches are way way better than symbolic approaches. Given almost any problem, it is much much easier to make an ML approach work than any symbolic approach. Yes, chess was first solved symbolically, but it's since been solved by ML better and more easily, to the point that stockfish no…

> I would challenge you to name any (non-simple) problem where traditional AI methods are still state of the art. Did ML methods best classic AI in dialog comprehension, say to the level of SHRDLU? I'm curious, can ML system do that - https://en.wikipedia.org/wiki/SHRDLU ?

Short answer: no. The capabilities of SHRDLU remain unsurpassed by modern systems.

A simulated robot hand controlled by natural language to move blocks inside a virtual world. It's not terribly useful but nothing that was created since can do any better and the state-of-the-art NLP approach of large language models is completely incapable of anything like it.

Which is a bit sad, really, if you think that SHRDLU was created by one graduate student, fifty years ago.

Post reply on HN