Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI winter is well on its way

blog.piekniewski.info

Post reply on HN
Print view View original

331–340 of 518 posts

Re: AI winter is well on its way

#331

Post by dpwm »

norswap wrote: post #302
joe_the_user wrote: post #81

This is a deep, significant post (pardon pun etc). The author is clearly informed and takes a strong, historical view of the situation. Looking at what the really smart people who brought us this innovation have said and done lately is a good start imo (just one datum of course, but there are others in this interesting survey). Deepmind hasn't shown anything breathtaking since their Alpha Go zero. Another thing to co…

Bleh, no it isn't. I am 100% in agreement with the author on the thesis: deep learning is overhyped and people project too much. But the content of the post is in itself not enough to advocate for this position. It is guilty of the same sins: projection and following social noises. The point about increasing compute power however, I found rather strong. New advances came at a high compute cost. Although it could be s…

> A much stronger rebuttal of the hype would have been based on the technical limitations of deep learning.

I'm not even sure how you'd go about doing that. You could use information theory to debunk some of the more ludicrous claims, especially ones that involve creating "missing" information.

One of the things that disappoints me somewhat with the field, which I've arguably only scratched the surface of, is just how much of it is driven by headline results which fail to develop understanding. A lot of the theory seems to be retrofitted to explain the relatively narrow result improvement and seems only to develop the art of technical bullshitting.

There are obvious exceptions to this and they tend to be the papers that do advance the field. With a relatively shallow resnet it's possible to achieve 99.7% on MNIST and 93% on CIFAR10 on a last-gen mid-range GPU with almost no understanding of what is actually happening.

There's also low-hanging fruit that seems to have been left on the tree. Take OpenAI's paper on parametrization of weights, so that you have a normalized direction vector and a scalar. This makes intuitive sense for anybody familiar with high-dimensional spaces since nearly all of the volume of a hypersphere lies around the surface. That this works in practice is great news, but leaves many questions unanswered.

I'm not even sure how many practitioners are thinking in high dimensional spaces or aware of their properties. It feels like we get to the universal approximation theorem and just accept that as evidence that they'll work well anywhere and then just follow whatever the currently recognised state of the art model is and adapt that to our purposes.

Re: AI winter is well on its way

#332

Post by raesene9 »

jacksmith21006 wrote: post #329
felippee wrote: post #15

Earlier quoted context omitted.

Hi, author here: Well first off: letters to investors are among the most biased pieces of writing in existence. Second: I'm not saying connectionism did not succeed in many areas! I'm a connectionist by heart! I love connectionism! But that being said there is disconnect between the expectations and reality. And it is huge. And it is particularly visible in autonomous driving. And it is not limited to media or CEO's,…

" letters to investors are among the most biased pieces of writing in existence. " Maybe true but they are words that are about things which are either true or not true. Has nothing to do where the words were shared. Saying they are on an investment letter so not relevant seems very short sighted. But just looking at the last 12 months it is folly to say we are moving to a AI winter. Things are just flying. Look at s…

Of course Google (or any other company) aren't going to blatently lie in a letter to investors (that kind of thing gets you sued) but it's pretty easy to spin words to sound more impressive than they may actually be.

Using the list provided, one example

"caption over a billion videos in 10 languages on YouTube;" - This doesn't say how accurate the captions acutally are. In my experience youtube captioning even of english dialect isn't exactly great. For one example try turning on the captions on this https://www.youtube.com/watch?v=bQJrBSXSs6o

so it's true I'm sure to say they've captioned the videos AI based techniques, but that doesn't mean they're a perfected option.

Also (purely anecodtally) Google translate also isn't exactly perfect yet either...

Re: AI winter is well on its way

#333

Post by jdonaldson »

zerostar07 wrote: post #312

An AI "winter" is a long period in which [edit: funding is cut because...] researchers are in disbelief about having a path to real intelligence. I think that is not the case at this time, because we have (or approaching) adequate tools to rationally dismiss that disbelief. The current AI "spring" has brought back the belief that connectionism may by the ultimate tool to explain the human brain. I mean you can't deny…

The brain has a number of functional parts that we don't understand all that well. Research on the brain hits a wall every now and then, but you never hear the phrase "Neuroscience Winter".

We're starting to train models that match biological brain behavior, at least in some crude functional/structural sense. Maybe this is just a string of coincidences, but my guess is that discoveries of analogous biological/model components will continue to happen, and we'll be able to learn more about AI and the brain by linking related phenomenon in vitro and in silica.

A few more recent examples:

https://deepmind.com/blog/grid-cells/

https://www.nature.com/articles/nature04485

Re: AI winter is well on its way

#334

Post by notahacker »

Erlich_Bachman wrote: post #204
canoebuilder wrote: post #163

Earlier quoted context omitted.

I think for most people, the research interest in games of various sorts, is not simply a desire for a better and better game contraption, a better mousetrap. But rather the thinking is, "playing games takes intelligence, what can we learn about intelligence by building machines that play games?" Most games are also closed systems, and conveniently grokkable systems, with enumerable search spaces. Which gives us easi…

This is so true, I can't understand why people miss this. The games are just games. It's intelligence that is the goal. And comparing Alpha Go Zero against those "other chess programs that existed for 30 years" is exactly missing the point also. Those programs were not constructed with zero-knowledge. They were carefully crafted by human players to achieve the result. Are we also going to count in all the brain proce…

The problem is that outside perfect information games, most areas where intelligence is required have few obvious routes to allow the computer to learn by perfectly simulating strategies and potential outcomes. Cases where "intelligence" is required typically entail handling human approximations of a lot of unknown and barely known possibilities with an inadequate dataset, and advances in approaches to perfect information games which can be entirely simulated by a machine knowing the ruleset (and possibly actually perturbed by adding inputs of human approaches to the problem) might be at best orthogonal to that particular goal. One of the takeaways from AlphaGo Zero massively outperforming AlphaGo is that even very carefully designed training sets for a problem fairly well understood by humans might actually retard system performance...

Re: AI winter is well on its way

#335

Post by zerostar07 »

jdonaldson wrote: post #333
zerostar07 wrote: post #312

An AI "winter" is a long period in which [edit: funding is cut because...] researchers are in disbelief about having a path to real intelligence. I think that is not the case at this time, because we have (or approaching) adequate tools to rationally dismiss that disbelief. The current AI "spring" has brought back the belief that connectionism may by the ultimate tool to explain the human brain. I mean you can't deny…

The brain has a number of functional parts that we don't understand all that well. Research on the brain hits a wall every now and then, but you never hear the phrase "Neuroscience Winter". We're starting to train models that match biological brain behavior, at least in some crude functional/structural sense. Maybe this is just a string of coincidences, but my guess is that discoveries of analogous biological/model c…

Neuroscience has been in constant winter, mainly because the methods are too crude and small scale, and that by observing 100 neurons out of billions makes it impossible to tell the whole story. The deep mind papers are interesting, but they are just a start imho. It may be that they are focusing on a mere coincidence. Nevertheless it is exciting to see progress in that direction, thats why i don't think DL research is going to lose steam soon. It would require a major show-stopper discovery (like the minsky-papert paper). Tesla crashes are no such thing.

It seems CS people are sick and tired of the hype, but i feel neuroscientists are now warming up to it.

Re: AI winter is well on its way

#336

Post by ptero »

felippee wrote: post #15
nopinsight wrote: post #14

A different take by Google’s cofounder, Sergey Brin, in his most recent Founders’ Letter to investors: “The new spring in artificial intelligence is the most significant development in computing in my lifetime.” He listed many examples below the quote. “understand images in Google Photos; enable Waymo cars to recognize and distinguish objects safely; significantly improve sound and camera quality in our hardware; und…

Hi, author here: Well first off: letters to investors are among the most biased pieces of writing in existence. Second: I'm not saying connectionism did not succeed in many areas! I'm a connectionist by heart! I love connectionism! But that being said there is disconnect between the expectations and reality. And it is huge. And it is particularly visible in autonomous driving. And it is not limited to media or CEO's,…

Making cars that drive safely no current, busy roads is a very difficult task. It is not surprising that the current systems do not do that (yet). It is surprising to me how well they still do. The fact though that my phone understands my voice and my handwriting and does on the fly translation of menus and simple requests is a sign of a major progress, too.

AI is overhyped and overfunded at the moment, which is not unusual for a hot technology (synthetic biology; dotcoms). Those things go in cycles, but the down cycles are seldom all out winters. During the slowdowns best technologies still get funding (less lavish, but enough to work on) and one-hit wonders die, both of which is good in the long run. My friends working in biology are doing mostly fine even though there are no longer "this is the century of synthetic biology" posters at every airport and in every toilet.

Re: AI winter is well on its way

#337

Post by icc97 »

DonaldFisk wrote: post #277
icc97 wrote: post #265

Earlier quoted context omitted.

Big Blue relied on humans to do all the training. Alpha Go zero didn't need humans at all to do the training. That's a pretty major shift for humanity.

It's Deep Blue, not Big Blue. The parameters used by its evaluation function were tuned by the system on games played by human masters. But it's a mistake to think that a system learning by playing against itself is something new. Arthur Samuel's draughts (chequers) program did that in 1959.

Sorry, mix up, thanks for the correction.

It's not that it's new, it's that they've achieved it. Chess was orders of magnitude harder than draughts. The solution for draughts didn't scale to chess but Alpha Go zero showed that chess was ridiculously easy for it once it had learned Go.

Re: AI winter is well on its way

#338

Post by icc97 »

sangnoir wrote: post #313
DonaldFisk wrote: post #277

Earlier quoted context omitted.

It's Deep Blue, not Big Blue. The parameters used by its evaluation function were tuned by the system on games played by human masters. But it's a mistake to think that a system learning by playing against itself is something new. Arthur Samuel's draughts (chequers) program did that in 1959.

> It's Deep Blue, not Big Blue. Big Blue is fine - it's referring to the company and not the machine. From Wikipedia "Big Blue is a nickname for IBM"

I meant Deep Blue, but yeah Deep Blue was a play on Big Blue.

Re: AI winter is well on its way

#339

Post by fvdessen »

lispm wrote: post #276
crummy wrote: post #241

Earlier quoted context omitted.

Google Translate is way, way better than it used to be (at least German > English which I suppose is probably an easier task than many languages).

Google Translate of Goethe: Have now, ah! Philosophy, Law and medicine, And unfortunately also theology Thoroughly studied, with great effort. Here I am, I poor gate! And I'm as smart as before word for word not completely bad, but then it breaks when we have to translate 'Tor'. Google Translate is clueless because it is unable to derive that here the 'fool' is meant. It's unable to 'understand' that 'I poor gate' ma…

On the other hand Deepl gives translations for news articles that are of such quality that it allows me to read international news as if it were local. Definitely useful.

Re: AI winter is well on its way

#340

Post by xamuel »

sanxiyn wrote: post #197
nopinsight wrote: post #77

Earlier quoted context omitted.

Because there is a dearth of experts and a lack of deep technical knowledge among many business people, there are still a great many companies that have not yet started investing in deep learning or AI despite potential profits based on current technology. Non-tech sectors of the economy are probably underinvesting at the moment. This is analogous to the way electricity took decades to realize productivity gains in t…

While I agree there is underinvestment in non-tech sectors, I don't see why that would change and they will use deep learning. There are lots of profitable things in non-tech sectors that can be done with linear regression but not done.

This. And at the end of the day, deep learning is just a more sophisticated version of linear regression. (To listen to some people talking, you'd think if a machine just curve-fits enough data-points, it'll suddenly wake up and become self-aware or something? The delusion is just unbelievable!)
Post reply on HN