Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI winter is well on its way

blog.piekniewski.info

Post reply on HN
Print view View original

431–440 of 518 posts

Re: AI winter is well on its way

#431

Post by dontreact »

The discussion on radiology is extremely sloppy.

Andrew Ng claimed human level performance on one radiology task (pneumonia). This claim seems to hold up pretty well as far as I can tell. Then the person criticizing him on twitter posts results on a completely different set of tasks which are just baseline results in order to launch a competition. These results are already close to human level performance, and after the competition it's very possible they will exceed human level performance.

Yes it's true that doing well at only Pneumonia doesn't mean that the nets are ready to replace radiologists. However, it does mean that we now have reason to think that all of the other tasks can be conquered in a reasonably short time frame such that someone going into the field should at least consider how AI is going to shape the field going forward.

Re: AI winter is well on its way

#432

Post by yourapostasy »

moistoreos wrote: post #399

I know this about the state of Deep Learning but I like to point out: While autonomous driving systems aren't perfect, statistically they are much better at driving than humans. Tesla's autonomous system has had, what, 3 or 4 fatal incidents? Out of the thousands of cars on the road that's less than 0.001%. There will always be a margin of error in systems engineered by man, just hopefully moving forward fewer and fe…

Depending upon your statistical sources, US traffic fatalities are around 1.25-1.50 per 100 million miles. [1] All forms of real-world autonomous driving across all manufacturers across the world are still somewhere below 200 million miles, conservatively estimated. [2] [3] Between the Tesla and Uber fatalities, by these rough back-of-the-envelope numbers, autonomous of various grades is still roughly 2X higher than human drivers. Maybe 1X if you squint at the numbers hard enough, but likely not orders of magnitude lower. I don't anticipate rapid legislative and insurance liability protections for autonomous systems until we see orders of magnitude differences on a per 100 million miles driven basis, and that will take time.

Waymo racks up about 10,000 miles per day across about 600 vehicles spread in about 25 cities. [4] Roughly 3.6 million miles per year if they stay level, but they're anticipated to rapidly add more vehicles to their fleet. In the US alone, about 3.22 trillion miles were driven in 2016. [5] Don't know what a statistically valid sample size is based upon that (I get nonsensical results below 2000 miles, so I'm doing something stupid), though. If Waymo puts two orders of magnitude more cars out there, they'll still "only" rack up about 365 million miles per year, and not all the miles on the same version of software.

[1] https://en.wikipedia.org/wiki/Transportation_safety_in_the_U...

[2] https://www.theverge.com/2016/5/24/11761098/tesla-autopilot-...

[3] https://www.theverge.com/2017/5/10/15609844/waymo-google-sel...

[4] https://medium.com/waymo/waymo-reaches-5-million-self-driven...

[5] https://www.npr.org/sections/thetwo-way/2017/02/21/516512439...

Re: AI winter is well on its way

#433

Post by placebo »

ujal wrote: post #389
bitL wrote: post #348

I was recently "playing" with some radiology data. I had no chance to identify diagnoses myself with untrained eyes, something that probably takes years for a decent radiologist to master. Just by using DenseNet-BC-100-12 I ended up with 83% ROC AUC after a few hours of training. In 4 out of 12 categories this classifier beat best human performing radiologists. Now the very same model with no other change than adjust…

I don't get the sentiment of the article either. I can't speak for researchers but software engineers are living through very exciting times. State of the art in numbers: Image Classification - ~$55, 9hrs (ImageNet) Object Detection - ~$40, 6hrs (COCO) Machine Translation - ~$40, 6hrs (WMT '14 EN-DE) Question Answering - ~$5, 0.8hrs (SQuAD) Speech recognition - ~$90, 13hrs (LibriSpeech) Language Modeling - ~$490, 74h…

My thoughts on AGI (at least in the sense of being indistinguishable from interaction with a human) are the same as my thoughts on extraterrestrial life: I'll believe it only when I see it (or at least when provided with proof that the mechanism is understood). This extrapolation on a sample size of one is something I don't understand. How is the fact that machine learning can do specific stuff better than humans different in principle than the fact that a hand calculator can do some specific stuff better than humans? On what evidence can we extrapolate from this to AGI?

We haven't found life outside this planet, and we haven't created life in a lab, therefore n=1 for assessing probability of life outside earth (which means we can't calculate a probability for this yet). Likewise, we haven't created anything remotely like animal intelligence (let alone human) and we have no good theory regarding how it works, so n=1 for existing forms of general intelligence.

Note that I'm not saying there can be no extraterrestrial life or that we will never develop AGI, just that I haven't seen any evidence at this point in time that any opinions for or against their possibility are anything more than baseless speculation.

Re: AI winter is well on its way

#434

Post by dogprez »

sah2ed wrote: post #405
varelse wrote: post #394

Earlier quoted context omitted.

Duplex was impressive but cheap street magic: https://medium.com/@Michael_Spencer/google-duplex-demo-witch... Microsoft OTOH quietly shipped the equivalent in China last month: https://www.theverge.com/2018/5/22/17379508/microsoft-xiaoic... Google has lost a lot of steam lately IMO. Facebook is releasing better tools and Microsoft, the company they nearly vanquished a decade ago, is releasing better products. Google…

> Microsoft, the company they nearly vanquished a decade ago, is releasing better products. Google nearly vanquished Microsoft a decade ago? Where can I read more about this bit of history :) ? IMO, Axios [0] seem to do a better job of criticizing Google's Duplex AI claims, as they repeatedly reached out to their contacts at Google for answers. 0: https://www.axios.com/google-ai-demo-questions-9a57afad-9854...

I think they are overselling Google's contributions a bit. It was more "Web 2.0" that shook Microsoft's dominance in tech. Google was a big curator and pushed state-of-the-art. Google was built on a large network of commodity hardware, they were able to do that because of the Open Source Software. Microsoft licensing would have been prohibitive to such innovation. There was some reenforcement that helped Linux gain momentum in other domains like Mobile and Desktop. Googled helped curate "Web 2.0" with developments / acquisitions like Maps and Gmail. When more of your life was spent on the web, the operating system meant less and that's also why Apple was able to make strides with their platforms. People weren't giving up as much when they switched to Mac as they would have previously.

Microsoft was previously the gatekeeper to almost every interaction with software (roughly 1992 - 2002). I don't know of good books on it but Tim O'Reilly wrote quite a bit about Web 2.0.

Re: AI winter is well on its way

#435

Post by riku_iki »

bitL wrote: post #400
varelse wrote: post #392

Earlier quoted context omitted.

I'd clarify that there is a specific delusion that any data scientist straight out of some sort of online degree program can go toe to toe with the likes of Andrej Karpathy or David Silver with the power of "teh durp lurnins'." And the predictable disappointment arising from the craptastic shovelware they create is what's finally creating the long overdue disappointment. Further, I have repeatedly heard people who sh…

> Deep Learning makes feature engineering obsolete I think even if you avoid constructing features, you are basically doing a similar process where a single change in a hyper-parameter can have significant effects: - internal structure of a model (what types of blocks are you using and how do you connect them, what are they capable of together, how do gradients propagate?) - loss function (great results come only if…

> if you have a cluster of machines and a way to orchestrate meta-optimization of slightly changed models

curious, if there is any good quality open source project for this..

Re: AI winter is well on its way

#436

Post by Florin_Andrei »

khawkins wrote: post #387
bitL wrote: post #348

I was recently "playing" with some radiology data. I had no chance to identify diagnoses myself with untrained eyes, something that probably takes years for a decent radiologist to master. Just by using DenseNet-BC-100-12 I ended up with 83% ROC AUC after a few hours of training. In 4 out of 12 categories this classifier beat best human performing radiologists. Now the very same model with no other change than adjust…

AI winters are a result of a massive disparity between the expectations of the general public and the reality of where the technology currently sits. Just like an asset bubble, the value of the industry as a whole pops as people collectively realize that AI, while not being worthless, is worth significantly less than they thought. Understand that in pop-sci circles over the past several years the general public is be…

> AI winters are a result of a massive disparity between the expectations of the general public and the reality of where the technology currently sits.

I think they also happen when the best ideas in the field run into the brick wall of insufficiently developed computer technology. I remember writing code for a perceptron in the '90s on an 8 bit system, 64 k RAM - it's laughable.

But right now compute power and data storage seem plentiful, so rumors of the current wave's demise appear exaggerated.

Re: AI winter is well on its way

#437

Post by bitL »

whatshisface wrote: post #397
salawat wrote: post #393

Earlier quoted context omitted.

Alright. What is the cost of a false positive in that case? The problem AI runs into is that with too much faith in the machine, people STOP thinking and believe the machine. Where you might get a .44 detection rate on radiology data alone, that radiologist with a .39 or a doctor can consult alternate streams of information. The AI may still be helpful in reinforcing a decision to continue scrutinizing a set of probl…

The problem could be fixed by asking doctors to put their diagnosis into the machine before the machine reveals what it thinks. Then, a simple Bayesian calculation could be performed based on the historical performance of that algorithm, all doctors, and that specific doctor, leading to a final number that would be far more accurate. All of the thinking would happen before the device polluted the doctor's cognitve bi…

There is a problem with that approach that at some point hospital management starts rating doctors by how well their diagnoses match those automated ones, and punish those who deviate too much, removing any incentives to be better/different. I wouldn't underestimate this, dysfunctional management exhibits these traits in almost any mature business.

Re: AI winter is well on its way

#438

Post by sah2ed »

dogprez wrote: post #434
sah2ed wrote: post #405

Earlier quoted context omitted.

> Microsoft, the company they nearly vanquished a decade ago, is releasing better products. Google nearly vanquished Microsoft a decade ago? Where can I read more about this bit of history :) ? IMO, Axios [0] seem to do a better job of criticizing Google's Duplex AI claims, as they repeatedly reached out to their contacts at Google for answers. 0: https://www.axios.com/google-ai-demo-questions-9a57afad-9854...

I think they are overselling Google's contributions a bit. It was more "Web 2.0" that shook Microsoft's dominance in tech. Google was a big curator and pushed state-of-the-art. Google was built on a large network of commodity hardware, they were able to do that because of the Open Source Software. Microsoft licensing would have been prohibitive to such innovation. There was some reenforcement that helped Linux gain m…

My question was actually tongue-in-cheek, which I tried to communicate with the smiley face.

I'm quite familiar with Google's history and would not characterize them as having vanquished Microsoft.

For the most part, Microsoft doesn't need to lose for Google to win (except of course in the realm of web search and office productivity).

Re: AI winter is well on its way

#439

Post by pizza »

riku_iki wrote: post #435
bitL wrote: post #400

Earlier quoted context omitted.

> Deep Learning makes feature engineering obsolete I think even if you avoid constructing features, you are basically doing a similar process where a single change in a hyper-parameter can have significant effects: - internal structure of a model (what types of blocks are you using and how do you connect them, what are they capable of together, how do gradients propagate?) - loss function (great results come only if…

> if you have a cluster of machines and a way to orchestrate meta-optimization of slightly changed models curious, if there is any good quality open source project for this..

I'm not aware of actually written code/projects that does this, but try looking into neural architecture search, it should be useful https://github.com/markdtw/awesome-architecture-search

Re: AI winter is well on its way

#440

Post by bitL »

Florin_Andrei wrote: post #436
khawkins wrote: post #387

Earlier quoted context omitted.

AI winters are a result of a massive disparity between the expectations of the general public and the reality of where the technology currently sits. Just like an asset bubble, the value of the industry as a whole pops as people collectively realize that AI, while not being worthless, is worth significantly less than they thought. Understand that in pop-sci circles over the past several years the general public is be…

> AI winters are a result of a massive disparity between the expectations of the general public and the reality of where the technology currently sits. I think they also happen when the best ideas in the field run into the brick wall of insufficiently developed computer technology. I remember writing code for a perceptron in the '90s on an 8 bit system, 64 k RAM - it's laughable. But right now compute power and data…

I wish GPUs were 1000x faster... Then I could do some crazy magic with Deep Learning instead of waiting weeks for training to be finished...
Post reply on HN