Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Researchers: Are we on the cusp of an ‘AI winter’?

bbc.co.uk

Post reply on HN
Print view View original

91–100 of 123 posts

Re: Researchers: Are we on the cusp of an ‘AI winter’?

#91

Post by Piskvorrr »

drongoking wrote: post #66
jean- wrote: post #32

Earlier quoted context omitted.

> When the state-of-the-art is 97% on a task, there's only so much room for improvement. When models commonly achieve 97% on a task, it means it's time to define a harder task, as it's long stopped providing any useful signal.

I think when performance on a real world task can be expressed as a single percentage, we've over-simplified the hell out of it and it's time to rethink the problem.

Or worse, overfitted - which means the solution will implode when faced with real data (RIP EH).

Re: Researchers: Are we on the cusp of an ‘AI winter’?

#92

Post by Piskvorrr »

Veedrac wrote: post #23
rococode wrote: post #5

I think there's an interesting disconnect right now between research and practice. Cutting-edge research does feel like it's reaching a plateau - across most AI fields even "major" breakthroughs are only gaining a couple percentage points and we're probably starting to hit the limits of what current approaches can achieve. When the state-of-the-art is 97% on a task, there's only so much room for improvement. Yoav Gol…

> Cutting-edge research does feel like it's reaching a plateau It's really not. The second half of last year alone had MuZero and Megatron-LM, to name just a couple that most scream to me that we are actually progressing towards AGI. You say ‘When the state-of-the-art is 97% on a task’, but solved tasks are the least interesting tasks.

Do we have any other indicators that it's actually progressing somewhere, besides screaming? Even such a triviality as "how do we recognize that we got there"? The research is still in very early phases, IMNSHO: impressive practical applications appear, but they're side effects of what appears as random flailing: "build it bigger, see if it helps. Build it sideways, see of it helps. Build it at full moon, see if it helps".

That suggests that the applications are the low-hanging fruit, with far more interesting results still to be discovered.

Re: Researchers: Are we on the cusp of an ‘AI winter’?

#93

Post by Piskvorrr »

AlexCoventry wrote: post #29
cbanek wrote: post #19

> While AGI isn't going to be created any time soon, machines have learned how to master complex tasks like: > translating text into practically every language Note: they said they have mastered these tasks. Yeah... I'm not sure a lot of native speakers would agree. Here's a great example of using Google Translate to automatically translate a video game. https://www.youtube.com/watch?v=_uNkubEHfQU > Driving cars I'm…

> Now don't get me wrong, we've made a lot of progress, but I wonder if we can get these things to a place better than humans before the next economic recession. I don't think it's necessary to completely solve superhuman performance to achieve automation of great economic value. Some of the most famous AI achievements leverage a fairly modest intelligence improvement with massive amounts of classic automation. E.g.,…

They do. FTR, this is what killed EH: "computing...computing...computing...oh well, let's alert the human - so they can watch the crash about to happen in the next second."

Re: Researchers: Are we on the cusp of an ‘AI winter’?

#94

Post by YeGoblynQueenne »

richk449 wrote: post #76
YeGoblynQueenne wrote: post #75

Earlier quoted context omitted.

>> But the beauty of neural networks is that they can be very good at generalizing from a partial sample of the problem space. That is really not the case. Neural nets generalise very poorly, hence the need for ever larger amounts of data: to overcome their lack of generalisation by attempting to cover as many "cases" as possible. Edit: when this subject comes up I cite the following article, by François Chollet, mai…

It doesn’t need to generalize, just do sophisticated interpolation. Basing the results on a dense sampling of the input space is exactly what I was suggesting.

Apologies for the misunderstanding. You said "generalizing from a partial sample of the problem space" and I thought you meant generalisation to unseen data from few examples, which is generally what we would all like to get from machine learnig models (but don't).

But, if a neural net can't _extrapolate_ to unseen instances, I don't see how it can solve problems like the one you describe with any useful precision, again unless it's trained with gigantic amounts of examples (which you say is not required). And how is this reducing computational costs with respect to hand-coded solvers?

Re: Researchers: Are we on the cusp of an ‘AI winter’?

#95

Post by Piskvorrr »

cbanek wrote: post #19

> While AGI isn't going to be created any time soon, machines have learned how to master complex tasks like: > translating text into practically every language Note: they said they have mastered these tasks. Yeah... I'm not sure a lot of native speakers would agree. Here's a great example of using Google Translate to automatically translate a video game. https://www.youtube.com/watch?v=_uNkubEHfQU > Driving cars I'm…

As for the translations - even a seemingly innocent sentence like "the sand people ride single file to hide their numbers" is fraught with ambiguity. I wouldn't call "unable to tell file-the-tool from file-the-data-representation from file-the-line" mastery.

Useful for a rough meaning in a completely unfamiliar language, sure - the hyperhype surrounding current SoA is actually contributing to the descent to AI winter.

https://www.everything2.com/title/The+sand+people+ride+in+si...

Re: Researchers: Are we on the cusp of an ‘AI winter’?

#96

Post by FartyMcFarter »

xyhopguy wrote: post #72
dpflan wrote: post #68

Earlier quoted context omitted.

Do you know of any good resources to learn more about this idea of the rate of improvement of perception per percentage point?

https://en.wikipedia.org/wiki/Odds_ratio odds of crash a => 97% => 3 / 100 odds of crash for b => 99.99% => 1 / 10000 improvement (odds ratio in this case) is then 300x = odds of crash for b / odds of crash for a

That may be true at a given moment, but how about time? I usually don't care about the probability that I'll crash within the next millisecond, but I do care about the probability that I'll crash over a whole trip.

Re: Researchers: Are we on the cusp of an ‘AI winter’?

#97

Post by soVeryTired »

richk449 wrote: post #73

Why would there be an AI winter? Was there a car winter after cars became a growing product? Was there a processor winter after microprocessors became a growing product? ERP software? Didn’t the previous AI winter happen because the hardware wasn’t advanced enough to make the technology useful to most people? Since that is no longer the case, why this consistent belief that there will be another winter?

2001 wasn't exactly great for the semiconductor industry...

Re: Researchers: Are we on the cusp of an ‘AI winter’?

#98

Post by m0zg »

IMO, no. Unlike the last time, things actually work this time. Perceptual things especially. People in the thread seem to be dismissive of those "single digit percentage point" gains that are being made nearly every month in some important tasks, but those last few percentage points often decide whether the system is garbage or useful. Compare e.g. Siri and Google Assistant, for example. Likely relatively small difference on metrics which results in a _huge_ difference in usability.

Another mistake people make is they look at model performance on academic dataset and make unsubstantiated conclusions about usefulness of models. Guess what, practical tasks _do not_ involve academic datasets. Some of academic datasets are _stupid hard_ on purpose (e.g. ImageNet, which forces your net to recognize _dog breeds_ that few humans can recognize). If your problem is more constrained, and the dataset is large enough and clean enough, you can often get very good results on practical problems, even with models that do not do all that well in published research.

Re: Researchers: Are we on the cusp of an ‘AI winter’?

#99

Post by dr_dshiv »

We don't want AI, we want systems that work better autonomously. We have lots of autonomous systems, mostly run by people (a shop keeper for a shop owner, for instance). Now that we have reached certain limits of pure digital systems, more innovations (ie, changes leading to better system outcomes) will happen due to human involvement in the data understanding and automation. It's just going to look more like people going to work.

The idea that AI (ML models) would be designed once is silly. The tuning and application always involves human judgment over time. We just hide the human contributions to AI/ML systems because it gets too complicated. But really, all good/practicable/in-the-wild AI systems involve a lot of people-in-the-loop!

Re: Researchers: Are we on the cusp of an ‘AI winter’?

#100

Post by Veedrac »

Piskvorrr wrote: post #92
Veedrac wrote: post #23

Earlier quoted context omitted.

> Cutting-edge research does feel like it's reaching a plateau It's really not. The second half of last year alone had MuZero and Megatron-LM, to name just a couple that most scream to me that we are actually progressing towards AGI. You say ‘When the state-of-the-art is 97% on a task’, but solved tasks are the least interesting tasks.

Do we have any other indicators that it's actually progressing somewhere, besides screaming? Even such a triviality as "how do we recognize that we got there"? The research is still in very early phases, IMNSHO: impressive practical applications appear, but they're side effects of what appears as random flailing: "build it bigger, see if it helps. Build it sideways, see of it helps. Build it at full moon, see if it h…

MuZero is in some sense the proto-holy grail, in that it implements learning and planning into unstructured tasks over purely internal models. While there is an obvious chasm between it and the end point, this is still something that has only recently become more than an abstract goal, at least to any effect.

Being able to perform planning over ‘simple’ domains like Atari games and Go (and not even in the same trained model!) might not seem very comparable to the real thing, but evolutionary history spent the bulk of its time building up the basics—most animals fail most cognitive tasks—so I don't think this is indicative of the progress being misguided, especially given networks-on-GPUs is literally a 10 year old field.

I think MuZero is a clear example of building by principles over random flailing. I get why there does also seem to be the latter, but it's certainly not the whole of it, and anyhow it worked for evolution ;P.

Post reply on HN