Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI winter is well on its way

blog.piekniewski.info

Post reply on HN
Print view View original

391–400 of 518 posts

Re: AI winter is well on its way

#391

Post by goatlover »

epicmellon wrote: post #386

"it is striking that the system spent long seconds trying to decide what exactly is sees in front (whether that be a pedestrian, bike, vehicle or whatever else) rather than making the only logical decision in these circumstances, which was to make sure not to hit it." That is striking. It always sort of bothered me that AI is really a big conglomeration of many different concepts. What people are working on is deep l…

That's why Augmented Intelligence is a better term. It doesn't scare up visions of Skynet or Hal 9000 run amok. Nor does it promise utopian singularity right around the corner.

It just means better tools to increase human capacity. But it's not nearly as good at getting headlines in the media.

Re: AI winter is well on its way

#392

Post by varelse »

khawkins wrote: post #387
bitL wrote: post #348

I was recently "playing" with some radiology data. I had no chance to identify diagnoses myself with untrained eyes, something that probably takes years for a decent radiologist to master. Just by using DenseNet-BC-100-12 I ended up with 83% ROC AUC after a few hours of training. In 4 out of 12 categories this classifier beat best human performing radiologists. Now the very same model with no other change than adjust…

AI winters are a result of a massive disparity between the expectations of the general public and the reality of where the technology currently sits. Just like an asset bubble, the value of the industry as a whole pops as people collectively realize that AI, while not being worthless, is worth significantly less than they thought. Understand that in pop-sci circles over the past several years the general public is be…

I'd clarify that there is a specific delusion that any data scientist straight out of some sort of online degree program can go toe to toe with the likes of Andrej Karpathy or David Silver with the power of "teh durp lurnins'." And the predictable disappointment arising from the craptastic shovelware they create is what's finally creating the long overdue disappointment.

Further, I have repeatedly heard people who should know better, with very fancy advanced degrees, chant variants of "Deep Learning gets better with more data" and/or "Deep Learning makes feature engineering obsolete" as if they are trying to convince everyone around them as well as themselves that these two fallacious assumptions are the revealed truth handed down to mere mortals by the 4 horsemen of the field.

That said, if you put your ~10,000 hours into this, and keep up with the field, it's pretty impressive what high-dimensional classification and regression can do. Judea Pearl concurs: https://www.theatlantic.com/technology/archive/2018/05/machi...

My personal (and admittedly biased) belief is that if you combine DL with GOFAI and/or simulation, you can indeed work magic. AlphaZero is strong evidence of that, no? And the author of the article in this thread is apparently attempting to do the same sort of thing for self-driving cars. I wouldn't call this part of the field irrational exuberance, I'd call it amazing.

Re: AI winter is well on its way

#393

Post by salawat »

bitL wrote: post #385
jeffreyrogers wrote: post #372

Earlier quoted context omitted.

I don't think the claim is that AI isn't useful. It's that it's oversold. In any case, I don't think you can tell much about how well your classifier is working for something like cancer diagnoses unless you know how many false negatives you have (and how that compares to how many false negatives a radiologist makes).

There are two sides to this: - how good humans are in detecting cancer (hint: not very good) and if having an automated system even as a "second opinion" next to an expert might not be useful? - there are metrics for capturing true/false positives/negatives one can focus on during learning optimization From studies you might have noticed that expert radiologists have e.g. F1-score at 0.45 and on average they score 0.…

Alright. What is the cost of a false positive in that case?

The problem AI runs into is that with too much faith in the machine, people STOP thinking and believe the machine. Where you might get a .44 detection rate on radiology data alone, that radiologist with a .39 or a doctor can consult alternate streams of information. The AI may still be helpful in reinforcing a decision to continue scrutinizing a set of problem.

AI's as we call them today are better referred to as expert systems. AI carries too much baggage to be thrown around Willy nilly. An expert system may beat out a human at interpreting large unintuitive datasets, but they aren't generally testable, and like it or not, it will remain a tough sell in any situation where lives are on the line.

I'm not saying it isn't worth researching, but AI will continue to fight an uphill battle in terms of public acceptance outside of research or analytics spaces, and overselling or being anything but straightforward about what is going on under the hood will NOT help.

Re: AI winter is well on its way

#394

Post by varelse »

ujal wrote: post #389
bitL wrote: post #348

I was recently "playing" with some radiology data. I had no chance to identify diagnoses myself with untrained eyes, something that probably takes years for a decent radiologist to master. Just by using DenseNet-BC-100-12 I ended up with 83% ROC AUC after a few hours of training. In 4 out of 12 categories this classifier beat best human performing radiologists. Now the very same model with no other change than adjust…

I don't get the sentiment of the article either. I can't speak for researchers but software engineers are living through very exciting times. State of the art in numbers: Image Classification - ~$55, 9hrs (ImageNet) Object Detection - ~$40, 6hrs (COCO) Machine Translation - ~$40, 6hrs (WMT '14 EN-DE) Question Answering - ~$5, 0.8hrs (SQuAD) Speech recognition - ~$90, 13hrs (LibriSpeech) Language Modeling - ~$490, 74h…

Duplex was impressive but cheap street magic: https://medium.com/@Michael_Spencer/google-duplex-demo-witch...

Microsoft OTOH quietly shipped the equivalent in China last month: https://www.theverge.com/2018/5/22/17379508/microsoft-xiaoic...

Google has lost a lot of steam lately IMO. Facebook is releasing better tools and Microsoft, the company they nearly vanquished a decade ago, is releasing better products. Google does remain the master of its own hype though.

Re: AI winter is well on its way

#395

Post by goatlover »

danielbarla wrote: post #244
kelnage wrote: post #228

Earlier quoted context omitted.

But software is starting from the same base. To claim it isn't would be to claim that the computers programmed themselves completely (which is simply not true).

Sure, there is some base there, and a fair bit of programming existed in the structure of the implementation. However, the heuristics themselves were not, and this is very significant. The software managed to reproduce and beat the previous best (both human and the previous iteration of itself), completely by playing against itself. So, in this sense, it's kind of like taking a human, teaching them the exact rules of…

> In my experience from chess, you'd be at a huge disadvantage if you started with this zero-knowledge handicap.

One problem is that we can't play millions of games against ourselves in a few hours. We can play a few games, grow tired, and then need to go do something else. Come back the next day, repeat. It's a very slow process, and we have to worry about other things in life. How much of one's time and focus can be used on learning a game? You could spend 12 hours a day, if you had no other responsibilities, I guess. That might be counter productive, though. We just don't have the same capacity.

If you artificially limited AlphaGo to human capacity, then my money would be on the human being a superior player.

Re: AI winter is well on its way

#396

Post by varelse »

mtgx wrote: post #358
dekhn wrote: post #324

I'm a scientist from a field outside ML who knows that ML can contribute to science. But I'm also really sad to see false claims in papers. For example, a good scientist can read an ML paper, see claims of 99% accuracy, and then probe further to figure out what the claims really mean. I do that a lot, and I find that accuracy inflation and careless mismanagement of data mars most "sexy" ML papers. To me, that's what'…

You hear Facebook all the time saying how it "automatically blocks 99% of the terrorist content " with AI to the public and governments. Nobody thought to ask: "How do you know all of that content is terrorist content? Does anyone check every video afterwards to ensure that all the blocked content was indeed terrorist content?" (assuming they even have an exact definition for it).

I'm 100% convinced it can block 99% of all terrorist content that hasn't been effectively SEOed to get around their filters because that's just memorizing attributes of the training set data. Unfortunately, the world isn't a stationary system like these ML models (usually) require. I still get spam in my gmail account, nowhere near as much as I do elsewhere, but I still get it.

Re: AI winter is well on its way

#397

Post by whatshisface »

salawat wrote: post #393
bitL wrote: post #385

Earlier quoted context omitted.

There are two sides to this: - how good humans are in detecting cancer (hint: not very good) and if having an automated system even as a "second opinion" next to an expert might not be useful? - there are metrics for capturing true/false positives/negatives one can focus on during learning optimization From studies you might have noticed that expert radiologists have e.g. F1-score at 0.45 and on average they score 0.…

Alright. What is the cost of a false positive in that case? The problem AI runs into is that with too much faith in the machine, people STOP thinking and believe the machine. Where you might get a .44 detection rate on radiology data alone, that radiologist with a .39 or a doctor can consult alternate streams of information. The AI may still be helpful in reinforcing a decision to continue scrutinizing a set of probl…

The problem could be fixed by asking doctors to put their diagnosis into the machine before the machine reveals what it thinks. Then, a simple Bayesian calculation could be performed based on the historical performance of that algorithm, all doctors, and that specific doctor, leading to a final number that would be far more accurate. All of the thinking would happen before the device polluted the doctor's cognitve biases.

Re: AI winter is well on its way

#398

Post by Maybestring »

twtw wrote: post #54
rch wrote: post #52

> Deepmind hasn't shown anything breathtaking since their Alpha Go zero. Didn't this just happen? Maybe my timescales are off, but I've been thinking about AI and Go since the late 90s, and plenty of real work was happening before then. Outside a handful of specialists, I'd expect another 8-10 years before the current state of the art is generally understood, much less effectively applied elsewhere.

I had the same response. AlphaZero was published like 5 months ago. Saying they've reached the end of the line because they haven't matched AlphaZero in six months is lame.

It also marked the end of a major multi-year project. With Deepmind moving that team to focus on other problems I wouldn't expect immediate results.

Re: AI winter is well on its way

#399

Post by moistoreos »

I know this about the state of Deep Learning but I like to point out:

While autonomous driving systems aren't perfect, statistically they are much better at driving than humans. Tesla's autonomous system has had, what, 3 or 4 fatal incidents? Out of the thousands of cars on the road that's less than 0.001%.

There will always be a margin of error in systems engineered by man, just hopefully moving forward fewer and fewer fatal ones.

Re: AI winter is well on its way

#400

Post by bitL »

varelse wrote: post #392
khawkins wrote: post #387

Earlier quoted context omitted.

AI winters are a result of a massive disparity between the expectations of the general public and the reality of where the technology currently sits. Just like an asset bubble, the value of the industry as a whole pops as people collectively realize that AI, while not being worthless, is worth significantly less than they thought. Understand that in pop-sci circles over the past several years the general public is be…

I'd clarify that there is a specific delusion that any data scientist straight out of some sort of online degree program can go toe to toe with the likes of Andrej Karpathy or David Silver with the power of "teh durp lurnins'." And the predictable disappointment arising from the craptastic shovelware they create is what's finally creating the long overdue disappointment. Further, I have repeatedly heard people who sh…

> Deep Learning makes feature engineering obsolete

I think even if you avoid constructing features, you are basically doing a similar process where a single change in a hyper-parameter can have significant effects:

- internal structure of a model (what types of blocks are you using and how do you connect them, what are they capable of together, how do gradients propagate?)

- loss function (great results come only if you use a fitting loss function)

- category weights (i.e. improving under-represented classes)

- image/data augmentation (self-driving car won't work without significant augmentation at all)

- properly set-up optimizer

The good thing here is that you can automate optimization of these to a large extent if you have a cluster of machines and a way to orchestrate meta-optimization of slightly changed models. With feature engineering you just have to do all the work upfront, thinking what might be important, and often you just miss important parts of features :-(

Post reply on HN