Live data from Hacker News

An understanding of AI’s limitations is starting to sink in

economist.com

291–300 of 403 posts

Re: An understanding of AI’s limitations is starting to sink in

#291
post #4

"The result is an artificial idiot savant that can excel at well-bounded tasks, but can get things very wrong if faced with unexpected input." I think this gets to the core of what is still a limitation of current technologies. Venturing into the unknown is still a deeply relevant task that seems unlikely to be replaced by computers anytime soon.

This is changing. In natural language processing just in the past couple weeks OpenAI wrote about their GPT3 model which can learn some tasks remarkably quickly, after only 1 or 2 examples. That model has extreme compute requirements, but it shows strong progress on performing never before seen tasks. There is still some steam left in the current deep learning boom. https://arxiv.org/abs/2005.14165

I heard someone theorize that GPT3 can perform so well simply because it can overfit to every domain

Re: An understanding of AI’s limitations is starting to sink in

#292

It's limited but so effective! The other day a friend asked for photos of her sister (also a friend) that I had because it's her birthday and she wanted to make a collage. I just searched on Google Photos by her name and it found a bunch because of the face classification. That's some good shit.

How do you know you haven't missed any good pictures of your sister ?

Re: An understanding of AI’s limitations is starting to sink in

#293

The trouble is that people have been sold this idea that ML/AI can do amazing things, without properly being told that really the things it can do are quite narrowly-scoped. They've been sold the Star Trek computer idea. For example, years ago I was working on a prototype/proof-of-concept thing for instrumenting industrial machinery with stick-on small computers. Simple stuff - attach accelerators, temperature, humid…

That's actually something ML is incredibly useful at, when it comes to machines with sensors - failure prediction / anomaly detection, etc. In the industry, (preventive) maintenance takes up a pretty huge chunk of resources. It's something techs need to do often, and it's often a laborious task, but it's obviously done to reduce downtime. So the business insight, as they like to call it, is to reduce costs tied up to…

If the business wanted to track the rate of failures and create predictive models about when things fail, or detect anomalous behaviour, that's what they would have set out with as the goal, and, perhaps, some ML model might have helped, but probably, it would've been too unreliable and any number of standard predictive models with well known characteristics would have been used instead.

That's not what they wanted.

What people are being sold is AI/ML as a magic bullet that will do something useful regardless of the situation, and it lets business people avoid making decisions about what they actually want, because AI/ML can be anything, so they just signup for it and expect to get 20 things they didn't know they wanted handed to them on a plate.

Turn out, it's not enough to just collect a bunch of data and wave your magic wand at it. It wasn't with web analytics 10 years ago, it's still not.

What you actually need is someone who has a bunch of tricks up their sleeve, and has done this before, and can suggest a bunch of Business Insights the business might need before they start building anything, people that actually decide what to do, and actions taken to investigate, and solve those problems.

I mean, to some degree you're right; perhaps ML models could be useful for tracking hardware failures, but that's not what the parent post is talking about. The previous post was talking about just collecting the data and expecting the predictive failure models to just jump out magically.

That doesn't happen; it needs a person to have the insight that the data could be used for such a thing, and that needs to happen before you go and randomly collect all the wrong frigging metrics.

...but hiring experts is expensive, and making decisions is hard. So ML/AI is sold like snake-oil to managers who want to avoid both of those things. :)

Re: An understanding of AI’s limitations is starting to sink in

#294

Earlier quoted context omitted.

That's actually something ML is incredibly useful at, when it comes to machines with sensors - failure prediction / anomaly detection, etc. In the industry, (preventive) maintenance takes up a pretty huge chunk of resources. It's something techs need to do often, and it's often a laborious task, but it's obviously done to reduce downtime. So the business insight, as they like to call it, is to reduce costs tied up to…

Yep, you and the user you're replying to are both right in different ways. One thing's for sure - machines don't generate "insights" on their own. Let's define an "insight" as "new meaningful knowledge", just for fun. We could talk about what comprises "new" and "meaningful" but it would be beside the point I'm making. In a supervised learning problem, the range of possible outputs is already known, meaning the model…

> Either you can only predict the types of failures you've already seen, or you can identify states you've never seen but you wouldn't know whether they mean the system is likely to fail soon or not.

Yup, this is something I've seen from both sides. First you mention is basically the standard, while the last is part of the deep learning voodoo black magic that executives and sales love.

I've had people approach me with proposals like "What if we just churn [ALL OF] our data through this or that model, and let's see if it comes up with some patterns we've never seen or thought about"

And that's not just for industrial applications. It's everywhere.

What is concerning to me is that this mentality will surely induce more unrealistic expectations. Before you know it, business execs are starting to ask why we need business analysts at all, because surely those fancy deep neural networks can extract all kinds of features - "only need data scientists to figure out those things".

So yeah, that's my fear. That businesses will blindly start to discard domain knowledge, and just feed black-box models their data, and let the data scientists wrestle with the results.

Re: An understanding of AI’s limitations is starting to sink in

#295
post #279
post #170

Earlier quoted context omitted.

Sure, ML is not at human-level intelligence - it is very much a tool-AI where we give a task to the machine and let it get very good at that. Nonetheless, it seems like progress on that front has been made incredibly quickly. Sure, Siri might not always "understand" what you're asking but the ML is able to very accurately transcribe your voice to text, something not really possible a few decades ago. Image classifier…

Progress in speech recognition has been slow and incremental, not fast and impressive. Hidden Markov Models did a decent job in the 90s. Now we have more data and more computing power.

In the past three years we have dropped error rates by a factor of 3, I don’t really think that claim holds water.

We have seen huge progress in a number of other fields as well. Anecdotally, voice recognition has definitely gotten way better as well.

Re: An understanding of AI’s limitations is starting to sink in

#296
post #16

Earlier quoted context omitted.

Absolutely. To be perfectly honest, it surprises me the extent to which ML naysaying seems to be popular on HN. The evidence of enormous progress seems pretty obvious to me.

> To be perfectly honest, it surprises me the extent to which ML naysaying seems to be popular on HN. Maybe because there is a larger fraction of people who know or see how the sausages are made. There are cool things, but is not magical nor transformative. At least not yet.

I work in “making the sausage” - deep learning techniques have already completely transformed my field and are transforming many others. Most of the people I see naysaying don’t appear to causally work in the field, because their critiques are not the same as the ones people in the field are using.

Re: An understanding of AI’s limitations is starting to sink in

#297

I'm not sure how anyone who's watched the exponential growth of a brand new domain can pick a point today to and say that things aren't as good as we expected. What may have happened was that some eager CEOs have overpromised on timelines and resources. But the revolution is coming, ML is already starting to change society. We're building the tech. Right now. The author does not even realise the immeasurable potentia…

> There's far more to do with ML and AI than self driving cars and shitty ad recommendations. Yeah, there's also shitty sentencing recommendations[1], new-age phrenology[2], and high-tech redlining[3]. I think your entire field needs to take a year off and take some ethics and philosophy courses before going any further. Otherwise we're all going to end up much worse off. [1] https://www.nytimes.com/2017/10/26/opinio…

This is a bit like telling the whole CS community in the 90s that we need a year off the internet to study ethics because people can use it for child porn, money laundering, and cybercrime.

Should we have?

Re: An understanding of AI’s limitations is starting to sink in

#298
post #236
post #96

Earlier quoted context omitted.

I completely agree with this. Let me add a non-US perspective that may surprise some people. I think a big part of what is holding back many companies from making effective, genuine, real-world use of AI is that a significant majority of the individuals involved are bad at their jobs . On the business side, there is an widespread unwillingness to acknowledge that technical people may be better placed to make decision…

Your argument boils down to “AI won’t be useful unless humans are twice as smart as they are” (your examples of the businesspeople and researchers), and thus doesn’t really say anything.

It is not smartness at all, but politics.

Re: An understanding of AI’s limitations is starting to sink in

#299

I'm not sure how anyone who's watched the exponential growth of a brand new domain can pick a point today to and say that things aren't as good as we expected. What may have happened was that some eager CEOs have overpromised on timelines and resources. But the revolution is coming, ML is already starting to change society. We're building the tech. Right now. The author does not even realise the immeasurable potentia…

Got bad news for you man. 16T in value doesn't come from ML. It comes from inflation and printing money out of thin air. The world has limited resources at the end of the day. When one industry does well, it's at the cost of a different industry.

Re: An understanding of AI’s limitations is starting to sink in

#300

Earlier quoted context omitted.

There has been next to ZERO progress towards genuine AGI despite a never-ending deluge of AI articles; that's normally the cause of scepticism. After several decades and a much-hyped last few years we have fake cleverness - impressively so in both cases - but nothing more.

Disagree. GPT-2/GPT-3 are able to pass for humans when the reader isn't paying close attention. This article seems insightful to me: http://www.overcomingbias.com/2017/03/better-babblers.html A scarily large amount of human speech is essentially word prediction, especially in cases where someone wants to seem impressive without having actually done the work. We're all familiar with the problem of people "bullshitting…

Where in the GPT models is the conscious (or even subconscious) thinking, comprehension, understanding, relating to experience, imagination etc, etc, you know: actual intelligence as a human would feel and use it, general or otherwise?

Tic Tac Toe, Chess, Image Classifying, Translation, Go, Sentence Construction, etc.; We are creating phenomenally impressive calculating worm-equivalents, but nothing human-level, or even remotely so.

Post reply on HN