Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI winter is well on its way

blog.piekniewski.info

Post reply on HN
Print view View original

411–420 of 518 posts

Re: AI winter is well on its way

#411

Post by varelse »

sah2ed wrote: post #405
varelse wrote: post #394

Earlier quoted context omitted.

Duplex was impressive but cheap street magic: https://medium.com/@Michael_Spencer/google-duplex-demo-witch... Microsoft OTOH quietly shipped the equivalent in China last month: https://www.theverge.com/2018/5/22/17379508/microsoft-xiaoic... Google has lost a lot of steam lately IMO. Facebook is releasing better tools and Microsoft, the company they nearly vanquished a decade ago, is releasing better products. Google…

> Microsoft, the company they nearly vanquished a decade ago, is releasing better products. Google nearly vanquished Microsoft a decade ago? Where can I read more about this bit of history :) ? IMO, Axios [0] seem to do a better job of criticizing Google's Duplex AI claims, as they repeatedly reached out to their contacts at Google for answers. 0: https://www.axios.com/google-ai-demo-questions-9a57afad-9854...

You're right, it was Steve Ballmer who nearly vanquished Microsoft at a time when Google was the company to work for in tech and kept doing amazing things. At least IMO.

Unfortunately, by the time of my brief stint at Google, the place was a professional dead-end where most of the hirees got smoke blown up their patooties at orientation about how amazing they were to be accepted into Google, only to be blind allocated into me-too MVPs of stuff they'd read about on TechCrunch. All IMO of course.

That said, I met the early Google Brain team there and I apparently made a sufficiently negative first impression for one of their leaders to hold a grudge against me 6 years later, explaining at last who it was that had blacklisted me there. So at least that mystery is solved.

PS It was pretty obvious these were voice actors in a studio conversing with the AI. That is impressive, but speaking as a former DJ myself, when one has any degree of voice training, one pronounces words without much accent and without slurring them together. Google will likely never admit anything here: they don't have to.

But I will give Alphabet a point for Waymo being the most professionally-responsible self-driving car effort so far. Compare and contrast with Tesla and Uber.

Re: AI winter is well on its way

#412

Post by fantasticsid »

mtgx wrote: post #358
dekhn wrote: post #324

I'm a scientist from a field outside ML who knows that ML can contribute to science. But I'm also really sad to see false claims in papers. For example, a good scientist can read an ML paper, see claims of 99% accuracy, and then probe further to figure out what the claims really mean. I do that a lot, and I find that accuracy inflation and careless mismanagement of data mars most "sexy" ML papers. To me, that's what'…

You hear Facebook all the time saying how it "automatically blocks 99% of the terrorist content " with AI to the public and governments. Nobody thought to ask: "How do you know all of that content is terrorist content? Does anyone check every video afterwards to ensure that all the blocked content was indeed terrorist content?" (assuming they even have an exact definition for it).

> Does anyone check every video afterwards to ensure that all the blocked content was indeed terrorist content?

They might not, but they could sample them to be statistically confident?

Re: AI winter is well on its way

#413

Post by eanzenberg »

bitL wrote: post #348

I was recently "playing" with some radiology data. I had no chance to identify diagnoses myself with untrained eyes, something that probably takes years for a decent radiologist to master. Just by using DenseNet-BC-100-12 I ended up with 83% ROC AUC after a few hours of training. In 4 out of 12 categories this classifier beat best human performing radiologists. Now the very same model with no other change than adjust…

ROC AUC is fairly useless when you have disparate costs in the errors. Try precision-recall.

Re: AI winter is well on its way

#414

Post by AstralStorm »

nl wrote: post #323
AstralStorm wrote: post #224

Earlier quoted context omitted.

Perfect information problem solving is not interesting anymore. Did they manage to extend it to games with hidden and imperfect information? (Say, chess with fog of war also known as Dark Chess. Phantom Go. Pathfinding equivalent would be an incremental search.) Edit: I see they are working on it, predictive state memory paper (MERLIN) is promising but not there yet.

There’s plenty of interesting work on poker bots.

Poker bots actually deal with a (simple) game with imperfect information. It is not the best test because short memory is sufficient to win at it.

The real challenge is to devise a general algorithm that will learn to be a good poker player in thousands of games, strategically, from just a bunch of games played. DeepStack AI required 10 million simulated games. Good human players outperform it at intermediate training stages.

And then the other part is figuring out actual rules of a harder game...

Re: AI winter is well on its way

#415

Post by [deleted] »

whatshisface wrote: post #397
salawat wrote: post #393

Earlier quoted context omitted.

Alright. What is the cost of a false positive in that case? The problem AI runs into is that with too much faith in the machine, people STOP thinking and believe the machine. Where you might get a .44 detection rate on radiology data alone, that radiologist with a .39 or a doctor can consult alternate streams of information. The AI may still be helpful in reinforcing a decision to continue scrutinizing a set of probl…

The problem could be fixed by asking doctors to put their diagnosis into the machine before the machine reveals what it thinks. Then, a simple Bayesian calculation could be performed based on the historical performance of that algorithm, all doctors, and that specific doctor, leading to a final number that would be far more accurate. All of the thinking would happen before the device polluted the doctor's cognitve bi…

[deleted]

Re: AI winter is well on its way

#416

Post by ASalazarMX »

bitL wrote: post #348

I was recently "playing" with some radiology data. I had no chance to identify diagnoses myself with untrained eyes, something that probably takes years for a decent radiologist to master. Just by using DenseNet-BC-100-12 I ended up with 83% ROC AUC after a few hours of training. In 4 out of 12 categories this classifier beat best human performing radiologists. Now the very same model with no other change than adjust…

> Just by using DenseNet-BC-100-12 I ended up with 83% ROC AUC after a few hours of training

Of course! Using DenseNet-BC-100-12 to increase ROC AUC, it was so obvious!

Re: AI winter is well on its way

#417

Post by bitL »

timr wrote: post #408
bitL wrote: post #348

I was recently "playing" with some radiology data. I had no chance to identify diagnoses myself with untrained eyes, something that probably takes years for a decent radiologist to master. Just by using DenseNet-BC-100-12 I ended up with 83% ROC AUC after a few hours of training. In 4 out of 12 categories this classifier beat best human performing radiologists. Now the very same model with no other change than adjust…

"Just by using DenseNet-BC-100-12 I ended up with 83% ROC AUC after a few hours of training." OK, but 83% ROC/AUC is nothing to be bragging about. ROC/AUC routinely overstates the performance of a classifier anyway, and even so, ~80% values aren't that great in any domain. I wouldn't trust my life to that level of performance, unless I had no other choice. You're basically making the author's case: deep learning clea…

Yeah, but the point here was that radiologists on average fared even worse. 83% is not impressive, but better than what we have right now in real-world with real people, as sad as it is. Obviously, best radiologists would outperform it right now, but average ones, likely stressed under heavy workload might not be able to beat it. And of course, this classifier probably works on certain visual structures better than humans and other ones easier detectable by humans would slip through.

There is also higher chance that next state-of-art model would push it significantly over 83% or best human radiologist at some point in the future, so it might not be very economical to train humans to become even better (i.e. dedicate your life to focus on radiology diagnostics only).

Re: AI winter is well on its way

#419

Post by letitgo12345 »

AI has a lot to offer to the industry right now I think where you don't need good worst case performance (ex., information retrieval, optimization, biology, etc.). The big problems in terms of application start appearing when you try and remove humans from the loop completely. That's not even close to possible yet but that doesn't mean the economic utility of even current AI is close to being maximized.

Re: AI winter is well on its way

#420

Post by [deleted] »

ASalazarMX wrote: post #416
bitL wrote: post #348

I was recently "playing" with some radiology data. I had no chance to identify diagnoses myself with untrained eyes, something that probably takes years for a decent radiologist to master. Just by using DenseNet-BC-100-12 I ended up with 83% ROC AUC after a few hours of training. In 4 out of 12 categories this classifier beat best human performing radiologists. Now the very same model with no other change than adjust…

> Just by using DenseNet-BC-100-12 I ended up with 83% ROC AUC after a few hours of training Of course! Using DenseNet-BC-100-12 to increase ROC AUC, it was so obvious!

[deleted]
Post reply on HN