Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI winter is well on its way

blog.piekniewski.info

Post reply on HN
Print view View original

41–50 of 518 posts

Re: AI winter is well on its way

#41

Post by evrydayhustling »

felippee wrote: post #34
oh-kumudo wrote: post #29

Author's reasons: 1.Hype dies down (which is really good! Meaning the chance of burst, is actually lower!) 2.Doesn't scale is false claim. DL methods have scaled MUCH better than any other ML algorithms in recent history (scale SVM is no small task). Scaling for DL methods are much either as comparing to other traditional ML algorithms, where it can be naturally distributed and aggregated. 3. Partially true. But self…

Author here: I'm using deep learning daily so I have a bit of an idea on what I'm talking about. 1) Not my point. Hype is doing very well. But narrative begins to crack, actually indicative of a burst... 2) DL does not scale very well. It does scale better than other ML algorithm because those did not scale at all. If you want to know what scales very well, look at CFD (computational fluid dynamics). DL in nowhere ne…

The scaling argument in the article doesn't make any sense. There are rhetorical queries like "does this model with 1000x as many parameters work 1000x as well?" but what it means to scale or perform are not clearly or consistently defined - let alone defined in a way that would make your point about the utility of the advances.

OpenAI's graph shows new architectures being used with more parameters because people are innovating on architecture and scale at the same time. Arguing that old methods "failed to scale" is like arguing that processor development was a failure because Intel had to develop a 486 instead of making a 386 work with more transistors (or more something).

And what does CFD have to do with anything, except maybe an odd attempt to argue from authority? Can you formalize from CFD a notion of "scaling well" well that anyone else agrees is useful for measuring AI research?

Re: AI winter is well on its way

#42

Post by rossdavidh »

felippee wrote: post #7
rossdavidh wrote: post #5

While I basically agree, really it ought to be called "AI autumn is well on its way", since I'm not sure we're into actual winter (i.e. dramatic reduction in $$ available for research) quite yet. But, probably soon.

Author here, yeah, it is the autumn. But I guess not many people would recognize the meaning, winter on the other hand is not ambiguous...

True.

Re: AI winter is well on its way

#43

Post by nopinsight »

felippee wrote: post #34
oh-kumudo wrote: post #29

Author's reasons: 1.Hype dies down (which is really good! Meaning the chance of burst, is actually lower!) 2.Doesn't scale is false claim. DL methods have scaled MUCH better than any other ML algorithms in recent history (scale SVM is no small task). Scaling for DL methods are much either as comparing to other traditional ML algorithms, where it can be naturally distributed and aggregated. 3. Partially true. But self…

Author here: I'm using deep learning daily so I have a bit of an idea on what I'm talking about. 1) Not my point. Hype is doing very well. But narrative begins to crack, actually indicative of a burst... 2) DL does not scale very well. It does scale better than other ML algorithm because those did not scale at all. If you want to know what scales very well, look at CFD (computational fluid dynamics). DL in nowhere ne…

What do you think of advances like those in major DeepMind papers? They seem to represent significant shifts in capabilities once matured.

Here's a recent example:

Unsupervised Predictive Memory in a Goal-Directed Agent

https://news.ycombinator.com/item?id=17177442

Re: AI winter is well on its way

#44

Post by oh-kumudo »

felippee wrote: post #34
oh-kumudo wrote: post #29

Author's reasons: 1.Hype dies down (which is really good! Meaning the chance of burst, is actually lower!) 2.Doesn't scale is false claim. DL methods have scaled MUCH better than any other ML algorithms in recent history (scale SVM is no small task). Scaling for DL methods are much either as comparing to other traditional ML algorithms, where it can be naturally distributed and aggregated. 3. Partially true. But self…

Author here: I'm using deep learning daily so I have a bit of an idea on what I'm talking about. 1) Not my point. Hype is doing very well. But narrative begins to crack, actually indicative of a burst... 2) DL does not scale very well. It does scale better than other ML algorithm because those did not scale at all. If you want to know what scales very well, look at CFD (computational fluid dynamics). DL in nowhere ne…

2)Why do you think DL doesn't scale? I am curious. It can easily leverage thousands of GPUs, training on 300 millions of images (https://ai.googleblog.com/2017/07/revisiting-unreasonable-ef...). No other methods is even close to leverage that amount of computational power. I don't really know about CFD, but at least in ML land and dealing with ML problems, DL is very scalable, maybe only next to random forests style algorithm, where they effectively share nothing.

3)It does matter. In fact most valuable startup around DL are CV based startups, they are mainly located in China though.

Re: AI winter is well on its way

#45

Post by baxtr »

nopinsight wrote: post #14

A different take by Google’s cofounder, Sergey Brin, in his most recent Founders’ Letter to investors: “The new spring in artificial intelligence is the most significant development in computing in my lifetime.” He listed many examples below the quote. “understand images in Google Photos; enable Waymo cars to recognize and distinguish objects safely; significantly improve sound and camera quality in our hardware; und…

Well, the way I see it: mostly, these are “improvements”, huge ones, but still. They ride the current AI tech wave, take it an optimize apps with it.

For most things, that people dream of and do marketing about need another leap forward, which we haven’t seen yet (it’ll come for sure)

Re: AI winter is well on its way

#46

Post by ericd »

buvanshak wrote: post #19
nopinsight wrote: post #14

A different take by Google’s cofounder, Sergey Brin, in his most recent Founders’ Letter to investors: “The new spring in artificial intelligence is the most significant development in computing in my lifetime.” He listed many examples below the quote. “understand images in Google Photos; enable Waymo cars to recognize and distinguish objects safely; significantly improve sound and camera quality in our hardware; und…

>caption over a billion videos in 10 languages on YouTube; Utterly useless. And I don't think it is improving.

I disagree, even with the high error rate, it provides a lot of context. Also, a lot of comedy.

Re: AI winter is well on its way

#47

Post by biswaroop »

bra-ket wrote: post #26
dhab wrote: post #20

Disclaimer: I am a lay technical person and don't know much about AI. I find this article somewhat condescending. I look at all the current development as stepping stones to progress, not an overnight success that does everything flawlessly. I imagine the future might be some combination of different solutions, and what the author proposes may or may not play a part in it.

it's not a stepping stone, if you look closely it's a dead end

I don't see how systematically accurate image classifiers and facial recognition systems built on deep learning is a 'dead end'. Products are products. If deep learning has led to actual profits in actual companies, it's not a dead end. As to whether this leads to AGI is a completely different question.

Re: AI winter is well on its way

#48

Post by prewett »

evrydayhustling wrote: post #41
felippee wrote: post #34

Earlier quoted context omitted.

Author here: I'm using deep learning daily so I have a bit of an idea on what I'm talking about. 1) Not my point. Hype is doing very well. But narrative begins to crack, actually indicative of a burst... 2) DL does not scale very well. It does scale better than other ML algorithm because those did not scale at all. If you want to know what scales very well, look at CFD (computational fluid dynamics). DL in nowhere ne…

The scaling argument in the article doesn't make any sense. There are rhetorical queries like "does this model with 1000x as many parameters work 1000x as well?" but what it means to scale or perform are not clearly or consistently defined - let alone defined in a way that would make your point about the utility of the advances. OpenAI's graph shows new architectures being used with more parameters because people are…

CFD was merely used as an example of something that does scale well. I'm not sure it was the best example, since CFD isn't very common. But basically you have a volume mesh and each cell iterates on the Navier-Stokes equation. So if you have N processor cores, you break the mesh in N pieces, each of which get processed in parallel. Doubling the number of cores allows you process double the amount in the same time, minus communication loses (each section of the mesh needs to communicate the results on its boundary to its neighbors).

I don't fully understand the graph, but it looks like his point is that Alpha Go Zero uses 1e5 times as many resources than AlexNet, but does not produce anywhere near 10,000 times better results. We saw that with CFDt 1e5 more cores resulted in 1e5 better results (= scales). The assertion is that DL's results are much less than 1e5 better, hence it does not scale.

Basically the argument is:

1. CFD produces N times better results given N times more resources [this is implied, requires a knowledge of CFD]. That is, f(ax) = a f(x). Or, f(ax) = 1 a * f(x).

2. Empirically, we see that DL has used 1e5 more resources, but is not producing 1e5 times better results. [No quantitative analysis of how much better the results are is given]

3. Since DL has f(a * x) = b * a * f(x), where b This isn't a very rigorous argument and the article left out half the argument, but it is suggestive.

Re: AI winter is well on its way

#49

Post by confounded »

tomatotomato37 wrote: post #13
thosakwe wrote: post #2

Truly, I agree. I've long been interested in learning about AI and deep learning, but to this day haven't done much that truly excites me within the field. It feels more or less impossible to make anything significant without Google-scale databases and Google-scale computers. AI really does make it easier for the few to jump far ahead, leaving everyone behind. I also agree that a lot the news around AI is just hype.…

I'm kinda curious now where those giant datasets will come from now that there's a big push for privacy with things like GDPR preventing some random researcher from just buying data off whatever data mining corp is most relevant to their AI's purpose

I’m kinda curious now which researchers have been buying PII from data mining corps.

Re: AI winter is well on its way

#50

Post by skybrian »

The argument is that self-driving won't work because Uber and Tesla had well-publicized crashes. But I don't see how this tells us anything about other, apparently more cautious companies like Waymo. There seem to be significant differences in technology.

More generally, machine learning is a broad area and there's no reason to believe that different applications of it will all succeed or all fail for similar reasons. It seems more likely there will be more winners along with many failed attempts.

Post reply on HN