AI winter is well on its way • Hacker News Board

Re: AI winter is well on its way

#421

Post by nmca » Wed, May 30, 2018, 4:22 PM UTC

dx034 wrote: post #281

MichaelMoser123 wrote: post #245

Forget about self driving cars - the real killer application of deep learning is mass surveillance - there are big customer for that (advertising, policing, political technology - we better get used to the term) and its the only technique that can get the job done. I sometimes think that there really was no AI winter as we got other technologies that implemented the ideas: SQL Databases can be seen as an application…

The BBC recently requested information about the use of facial recognition from UK police forces. Those that use facial recognition reported false positive rates of >95%. That led some to abandon the systems, others just use it as one form of pre-screening. Mass surveillance with facial recognition is nowhere near levels where it can be used unsupervised. And that's even before people actively try to deceive it. For…

Aha, yeah I saw this in the news - pretty classic use of people horribly misunderstanding statistics and/or misrepresenting the facts. Let's say one person out of 60 million has sauron's ring. I use my DeepMagicNet to classify everyone, and get 100 positive results. Only one is the ringbearer, so I have a 99% error rate. Best abandon ship.

Re: AI winter is well on its way

#422

Post by timr » Wed, May 30, 2018, 4:28 PM UTC

bitL wrote: post #417

timr wrote: post #408

Earlier quoted context omitted.

"Just by using DenseNet-BC-100-12 I ended up with 83% ROC AUC after a few hours of training." OK, but 83% ROC/AUC is nothing to be bragging about. ROC/AUC routinely overstates the performance of a classifier anyway, and even so, ~80% values aren't that great in any domain. I wouldn't trust my life to that level of performance, unless I had no other choice. You're basically making the author's case: deep learning clea…

Yeah, but the point here was that radiologists on average fared even worse. 83% is not impressive, but better than what we have right now in real-world with real people, as sad as it is. Obviously, best radiologists would outperform it right now, but average ones, likely stressed under heavy workload might not be able to beat it. And of course, this classifier probably works on certain visual structures better than h…

"Yeah, but the point here was that radiologists on average fared even worse."

Except they don't. See the table in the original post. Also, comparing the "average" radiologist by F1 scores from a single experiment (as you've done in other comments here) is meaningless.

Unless my doctor is exactly average (and isn't incorporating additional information, or smart enough to be optimizing for false positive/negative rates relative to cost), comparison to average statistics is academic. But I don't really need to tell you this -- your comment has so many caveats that you're clearly already aware of the limits of the method.

Re: AI winter is well on its way

#423

Post by bitL » Wed, May 30, 2018, 4:29 PM UTC

eanzenberg wrote: post #413

bitL wrote: post #348

I was recently "playing" with some radiology data. I had no chance to identify diagnoses myself with untrained eyes, something that probably takes years for a decent radiologist to master. Just by using DenseNet-BC-100-12 I ended up with 83% ROC AUC after a few hours of training. In 4 out of 12 categories this classifier beat best human performing radiologists. Now the very same model with no other change than adjust…

ROC AUC is fairly useless when you have disparate costs in the errors. Try precision-recall.

I mentioned F1 in some later comment.

Re: AI winter is well on its way

#424

Post by slow_donkey » Wed, May 30, 2018, 4:33 PM UTC

raesene9 wrote: post #332

jacksmith21006 wrote: post #329

Earlier quoted context omitted.

" letters to investors are among the most biased pieces of writing in existence. " Maybe true but they are words that are about things which are either true or not true. Has nothing to do where the words were shared. Saying they are on an investment letter so not relevant seems very short sighted. But just looking at the last 12 months it is folly to say we are moving to a AI winter. Things are just flying. Look at s…

Of course Google (or any other company) aren't going to blatently lie in a letter to investors (that kind of thing gets you sued) but it's pretty easy to spin words to sound more impressive than they may actually be. Using the list provided, one example "caption over a billion videos in 10 languages on YouTube;" - This doesn't say how accurate the captions acutally are. In my experience youtube captioning even of eng…

... I don't think I understand that video even with my own ears. YouTube captioning has actually significantly improved from it's previous hilarious state

Re: AI winter is well on its way

#425

Post by bitL » Wed, May 30, 2018, 4:35 PM UTC

timr wrote: post #422

bitL wrote: post #417

Earlier quoted context omitted.

Yeah, but the point here was that radiologists on average fared even worse. 83% is not impressive, but better than what we have right now in real-world with real people, as sad as it is. Obviously, best radiologists would outperform it right now, but average ones, likely stressed under heavy workload might not be able to beat it. And of course, this classifier probably works on certain visual structures better than h…

"Yeah, but the point here was that radiologists on average fared even worse." Except they don't. See the table in the original post. Also, comparing the "average" radiologist by F1 scores from a single experiment (as you've done in other comments here) is meaningless. Unless my doctor is exactly average (and isn't incorporating additional information, or smart enough to be optimizing for false positive/negative rates…

What would be the good metrics then? Of course metrics are just indicators that can be interpreted incorrectly. Still, we have to measure something tangible. What would you propose? I am aware of limitations and would gladly use something better...

Some people mention Matthews correlation coefficients, Youden's J statistic, Cohen's kappa etc. but I haven't seen them in any Deep Learning paper so far and I bet they have large blindspots as well.

Re: AI winter is well on its way

#426

Post by s-shellfish » Wed, May 30, 2018, 4:36 PM UTC

felippee wrote: post #403

Author here: seriously I'm here at the front page for the second day in the a row!? The sheer viral popularity of this post, which really was just a bunch of relatively loose thoughts indicates that there is something in the air regarding AI winter. Maybe people are really sick of all that hype pumping... Just a note: I'm a bit overwhelmed so I can't address all the criticism. One thing I would like to state however,…

Do you think there's a balance to be struck between connectionism and computationalism?

Re: AI winter is well on its way

#427

Post by iwintermute » Wed, May 30, 2018, 4:45 PM UTC

some_account wrote: post #242

felippee wrote: post #137

Earlier quoted context omitted.

It's not like humanity really needs another chess playing program 20 years after IBM solved that problem (but now utilizing 1000x more compute power). I just find all these game playing contraptions really uninteresting. There are plenty real world problems to be solved of much higher practicality. Moravec's paradox in full glow.

Tell me about it. The brightest minds are working on ads, and we have AI playing social games. Can AI make the world better? It can, but it won't since we are humans, and humans will weaponize technology every chance it gets. Of course some positive uses will come, but the negative ones will be incredibly destructive.

Let's start with defining "better"

Re: AI winter is well on its way

#428

Post by wqnt » Wed, May 30, 2018, 4:52 PM UTC

dx034 wrote: post #281

MichaelMoser123 wrote: post #245

Forget about self driving cars - the real killer application of deep learning is mass surveillance - there are big customer for that (advertising, policing, political technology - we better get used to the term) and its the only technique that can get the job done. I sometimes think that there really was no AI winter as we got other technologies that implemented the ideas: SQL Databases can be seen as an application…

The BBC recently requested information about the use of facial recognition from UK police forces. Those that use facial recognition reported false positive rates of >95%. That led some to abandon the systems, others just use it as one form of pre-screening. Mass surveillance with facial recognition is nowhere near levels where it can be used unsupervised. And that's even before people actively try to deceive it. For…

95% false positive rate is extremely good for surveillance, as the cost of false positive is low (wasted efforts by the police). That means for every 20 people the police investigate, one is a target.

Re: AI winter is well on its way

#429

Post by bitL » Wed, May 30, 2018, 4:54 PM UTC

timr wrote: post #422

bitL wrote: post #417

Earlier quoted context omitted.

Yeah, but the point here was that radiologists on average fared even worse. 83% is not impressive, but better than what we have right now in real-world with real people, as sad as it is. Obviously, best radiologists would outperform it right now, but average ones, likely stressed under heavy workload might not be able to beat it. And of course, this classifier probably works on certain visual structures better than h…

"Yeah, but the point here was that radiologists on average fared even worse." Except they don't. See the table in the original post. Also, comparing the "average" radiologist by F1 scores from a single experiment (as you've done in other comments here) is meaningless. Unless my doctor is exactly average (and isn't incorporating additional information, or smart enough to be optimizing for false positive/negative rates…

> Unless my doctor is exactly average

Just to get back to this point: what if the vision system of your doctor is below average and you augment her by giving her a statistically better vision system, while allowing her to use the additional sources as she sees fit. Wouldn't be that an improvement? We are talking about vision subsystem here, not the whole "reasoning package" human doctors posses.

Re: AI winter is well on its way

#430

Post by dgacmu » Wed, May 30, 2018, 5:05 PM UTC

AstralStorm wrote: post #410

dgacmu wrote: post #349

Earlier quoted context omitted.

Strongly disagree. There are a lot of approximation algorithms and heuristics in wide use - to the tune of trillions of dollars, in fact, when you consider transportation and logistics, things like asic place & route, etc. These are all intractable perfect info problems that are so widespread and commercially important that they amplify the effect of even modest improvements. (You said problems, not games...)

Indeed, there are a few problems where even with perfect information you will be hard pressed to solve them. But that is only a question of computational power or the issue when the algorithm does not allow efficient approximation (not in APX space or co-APX). The thing is, an algorithm that can work with fewer samples and robustly tolerating mistakes in datasets (also known as imperfect information) will be vastly c…

Perhaps a unifying way of saying this is: it's a research problem to figure out how to get ML techniques to the point they outperform existing heuristics on "hard" problems. Doing so will result in engineering improvements to the specific systems that need approximate solutions to those problems.

I completely agree about the importance of imperfect information problems. In practice, many techniques handle some label noise, but not optimally. Even MNIST is much easier to solve if you remove the one incorrectly-labeled training example. (one! Which is barely noise. Though as a reassuring example from the classification domain, JFT is noisy and still results in better real world performance than just training on imagenet.)