Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI winter is well on its way

blog.piekniewski.info

Post reply on HN
Print view View original

401–410 of 518 posts

Re: AI winter is well on its way

#401

Post by kinsomo »

khawkins wrote: post #387
bitL wrote: post #348

I was recently "playing" with some radiology data. I had no chance to identify diagnoses myself with untrained eyes, something that probably takes years for a decent radiologist to master. Just by using DenseNet-BC-100-12 I ended up with 83% ROC AUC after a few hours of training. In 4 out of 12 categories this classifier beat best human performing radiologists. Now the very same model with no other change than adjust…

AI winters are a result of a massive disparity between the expectations of the general public and the reality of where the technology currently sits. Just like an asset bubble, the value of the industry as a whole pops as people collectively realize that AI, while not being worthless, is worth significantly less than they thought. Understand that in pop-sci circles over the past several years the general public is be…

> But eventually the general public wises up to the unfulfillment of expectations, and drives their attention elsewhere. Here we have the AI winter.

And more importantly, business and government leaders wise up and turn off the money tap.

Re: AI winter is well on its way

#402

Post by varelse »

bitL wrote: post #400
varelse wrote: post #392

Earlier quoted context omitted.

I'd clarify that there is a specific delusion that any data scientist straight out of some sort of online degree program can go toe to toe with the likes of Andrej Karpathy or David Silver with the power of "teh durp lurnins'." And the predictable disappointment arising from the craptastic shovelware they create is what's finally creating the long overdue disappointment. Further, I have repeatedly heard people who sh…

> Deep Learning makes feature engineering obsolete I think even if you avoid constructing features, you are basically doing a similar process where a single change in a hyper-parameter can have significant effects: - internal structure of a model (what types of blocks are you using and how do you connect them, what are they capable of together, how do gradients propagate?) - loss function (great results come only if…

Yep, and in doing so, you just traded "feature engineering" for graph design and data prep, no? And that's my response to these sorts. And their usual response to me is to grumble that I don't know what I'm doing. I've started tuning them out of my existence because they seem to have nothing to contribute to it.

Re: AI winter is well on its way

#403

Post by felippee »

Author here: seriously I'm here at the front page for the second day in the a row!?

The sheer viral popularity of this post, which really was just a bunch of relatively loose thoughts indicates that there is something in the air regarding AI winter. Maybe people are really sick of all that hype pumping...

Just a note: I'm a bit overwhelmed so I can't address all the criticism. One thing I would like to state however, is that I'm actually a fan of connectionism. I think we are doing it naively though and instead of focusing on the right problem we inflate a hype bubble. There are applications where DL really shines and there is no question about that. But in case of autonomy and robotics we have not even defined the problems well enough, not to mention solving anything. But unfortunately, those are the areas where most best/expectations sit, therefore I'm worried about the winter.

Re: AI winter is well on its way

#404

Post by braindongle »

jeffreyrogers wrote: post #372
bitL wrote: post #348

I was recently "playing" with some radiology data. I had no chance to identify diagnoses myself with untrained eyes, something that probably takes years for a decent radiologist to master. Just by using DenseNet-BC-100-12 I ended up with 83% ROC AUC after a few hours of training. In 4 out of 12 categories this classifier beat best human performing radiologists. Now the very same model with no other change than adjust…

I don't think the claim is that AI isn't useful. It's that it's oversold. In any case, I don't think you can tell much about how well your classifier is working for something like cancer diagnoses unless you know how many false negatives you have (and how that compares to how many false negatives a radiologist makes).

Is there a prevailing approach to thinking about (accounting for?) false negatives in ground truth data? I'm new to this area, and the question is relevant for my current work. By definition, you simply don't know anything about false negatives unless you have some estimate of specificity in addition to your labeled data, but can anything be done?

Re: AI winter is well on its way

#405

Post by sah2ed »

varelse wrote: post #394
ujal wrote: post #389

Earlier quoted context omitted.

I don't get the sentiment of the article either. I can't speak for researchers but software engineers are living through very exciting times. State of the art in numbers: Image Classification - ~$55, 9hrs (ImageNet) Object Detection - ~$40, 6hrs (COCO) Machine Translation - ~$40, 6hrs (WMT '14 EN-DE) Question Answering - ~$5, 0.8hrs (SQuAD) Speech recognition - ~$90, 13hrs (LibriSpeech) Language Modeling - ~$490, 74h…

Duplex was impressive but cheap street magic: https://medium.com/@Michael_Spencer/google-duplex-demo-witch... Microsoft OTOH quietly shipped the equivalent in China last month: https://www.theverge.com/2018/5/22/17379508/microsoft-xiaoic... Google has lost a lot of steam lately IMO. Facebook is releasing better tools and Microsoft, the company they nearly vanquished a decade ago, is releasing better products. Google…

> Microsoft, the company they nearly vanquished a decade ago, is releasing better products.

Google nearly vanquished Microsoft a decade ago? Where can I read more about this bit of history :) ?

IMO, Axios [0] seem to do a better job of criticizing Google's Duplex AI claims, as they repeatedly reached out to their contacts at Google for answers.

0: https://www.axios.com/google-ai-demo-questions-9a57afad-9854...

Re: AI winter is well on its way

#406

Post by kinsomo »

salawat wrote: post #393
bitL wrote: post #385

Earlier quoted context omitted.

There are two sides to this: - how good humans are in detecting cancer (hint: not very good) and if having an automated system even as a "second opinion" next to an expert might not be useful? - there are metrics for capturing true/false positives/negatives one can focus on during learning optimization From studies you might have noticed that expert radiologists have e.g. F1-score at 0.45 and on average they score 0.…

Alright. What is the cost of a false positive in that case? The problem AI runs into is that with too much faith in the machine, people STOP thinking and believe the machine. Where you might get a .44 detection rate on radiology data alone, that radiologist with a .39 or a doctor can consult alternate streams of information. The AI may still be helpful in reinforcing a decision to continue scrutinizing a set of probl…

> The problem AI runs into is that with too much faith in the machine, people STOP thinking and believe the machine.

See https://youtu.be/R_rF4kcqLkI?t=2m51s

In medicine, I want everyone to apply appropriate skepticism to important results, and I don't want to enable lazy radiologists to zone out and press 'Y' all day. I want all the doctors to be maximally mentally engaged. Skepticism of an incorrect radiologist report recently saved my dad from some dangerous, and in his case unnecessary, treatment.

Re: AI winter is well on its way

#407

Post by scardine »

jimothywales wrote: post #205
scardine wrote: post #132

Earlier quoted context omitted.

The difference between Blockchain advocates and the thieves from the tale "The Emperor's New Clothes" is that the weavers from the tale knew they were bullshitting the king. Sadly blockchains will not revolutionize mankind.

That's not entirely correct. People repeatedly like to take hard yes/no stances on blockchain because they seem to confuse the issue with cryptocurrencies. Cryptocurrencies are quite obviously overhyped and self-destructing. But blockchain is actually being implemented in many effective ways around the world - just in areas that you wouldn't generally hear about. Blockchain has its place, just not at the current mome…

I don't know any problem where blockchains are the best solution - it is possible there are some, I don't know.

I'm just talking about the very inflated social/political expectations, the "blockchain will change everything" mantra: I don't see this happening.

Re: AI winter is well on its way

#408

Post by timr »

bitL wrote: post #348

I was recently "playing" with some radiology data. I had no chance to identify diagnoses myself with untrained eyes, something that probably takes years for a decent radiologist to master. Just by using DenseNet-BC-100-12 I ended up with 83% ROC AUC after a few hours of training. In 4 out of 12 categories this classifier beat best human performing radiologists. Now the very same model with no other change than adjust…

"Just by using DenseNet-BC-100-12 I ended up with 83% ROC AUC after a few hours of training."

OK, but 83% ROC/AUC is nothing to be bragging about. ROC/AUC routinely overstates the performance of a classifier anyway, and even so, ~80% values aren't that great in any domain. I wouldn't trust my life to that level of performance, unless I had no other choice.

You're basically making the author's case: deep learning clearly outperforms on certain classes of problems, and easily "generalizes" to modest performance on lots of others. But leaping from that to "radiology robots are almost here!" is folly.

Re: AI winter is well on its way

#409

Post by khawkins »

kinsomo wrote: post #401
khawkins wrote: post #387

Earlier quoted context omitted.

AI winters are a result of a massive disparity between the expectations of the general public and the reality of where the technology currently sits. Just like an asset bubble, the value of the industry as a whole pops as people collectively realize that AI, while not being worthless, is worth significantly less than they thought. Understand that in pop-sci circles over the past several years the general public is be…

> But eventually the general public wises up to the unfulfillment of expectations, and drives their attention elsewhere. Here we have the AI winter. And more importantly, business and government leaders wise up and turn off the money tap.

This is why it's wise for researchers and business leaders to temper expectations. Better a constant modest flow of money into the field than a boom-bust cycle with huge upfront investment followed by very bearish actions.

Re: AI winter is well on its way

#410

Post by AstralStorm »

dgacmu wrote: post #349
AstralStorm wrote: post #224

Earlier quoted context omitted.

Perfect information problem solving is not interesting anymore. Did they manage to extend it to games with hidden and imperfect information? (Say, chess with fog of war also known as Dark Chess. Phantom Go. Pathfinding equivalent would be an incremental search.) Edit: I see they are working on it, predictive state memory paper (MERLIN) is promising but not there yet.

Strongly disagree. There are a lot of approximation algorithms and heuristics in wide use - to the tune of trillions of dollars, in fact, when you consider transportation and logistics, things like asic place & route, etc. These are all intractable perfect info problems that are so widespread and commercially important that they amplify the effect of even modest improvements. (You said problems, not games...)

Indeed, there are a few problems where even with perfect information you will be hard pressed to solve them. But that is only a question of computational power or the issue when the algorithm does not allow efficient approximation (not in APX space or co-APX).

The thing is, an algorithm that can work with fewer samples and robustly tolerating mistakes in datasets (also known as imperfect information) will be vastly cheaper and easier to operate. Less tedious sample data collection and labelling.

Working with lacking and erroneous information (without known error value) is necessarily a crucial step towards AGI; as is extracting structure from such data.

This is the difference between an engineering problem and research problem.

Post reply on HN