Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to recognize AI snake oil [pdf]

cs.princeton.edu

Post reply on HN
Print view View original

141–150 of 364 posts

Re: How to recognize AI snake oil [pdf]

#141

Post by tensor »

nerdponx wrote: post #86
reggieband wrote: post #58

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…

That's surprising to hear. Comparing model performance to a randomized baseline model is a "must-have" on my team before we feel comfortable presenting to management.

Getting a team to publish their results (after patenting) is also a good way to get them to do these sorts of things. Significance, baselines, and other things are asked for by reviewers for the better journals and conferences.

Re: How to recognize AI snake oil [pdf]

#142

Post by tensor »

chimi wrote: post #125
formalsystem wrote: post #15

Over the years my heuristic has turned into: "Did the team formulate their problem as a supervised learning problem?" - If not it's probably BS. In longform if anyone is interested https://medium.com/@marksaroufim/can-deep-learning-solve-my-... EDIT: I would consider autoencoders, word2vec, Reinforcement Learning examples of turning a different problem into a supervised learning problem EDIT 2: Social functions like…

It's hard to verbalize this, most of it is "intuition" but I think it boils down to "supervised learning is BS." Humans are smarter than computers. How can a human teach a computer how to do something when the human itself can't teach another human that something? We haven't solved that problem. The snake is eating its tail. You can't teach a human how to do something when the methodology to do that is the student tr…

Supervised learning in machine learning is nothing remotely like a human teaching anyone anything. It's a very clear mathematical formulation of what the objective is and how the algorithm can improve itself against that objective.

The closest analogy for humans would be to define a metric and ask a human to figure out how to maximize that metric. That's something we're often pretty good at doing, often in ways that the person defining the metric didn't actually want us to use.

Re: How to recognize AI snake oil [pdf]

#143

Post by [deleted] »

reggieband wrote: post #58

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…

[deleted]

Re: How to recognize AI snake oil [pdf]

#144

Post by mark-r »

wilg wrote: post #137
chimi wrote: post #125

Earlier quoted context omitted.

It's hard to verbalize this, most of it is "intuition" but I think it boils down to "supervised learning is BS." Humans are smarter than computers. How can a human teach a computer how to do something when the human itself can't teach another human that something? We haven't solved that problem. The snake is eating its tail. You can't teach a human how to do something when the methodology to do that is the student tr…

Well... why is it necessary that we can teach a human to do something in order to teach a machine to do it?

Teaching a human is a heuristic for understanding the problem well enough to teach a machine.

Re: How to recognize AI snake oil [pdf]

#145

Post by dhairya »

From the slides:

Harms of AI for predicting social outcomes

• Hunger for personal data

• Massive transfer of power from domain experts & workers to unaccountable tech companies

• Lack of explainability

• Distracts from interventions

• Veneer of accuracy

Human behavior is not IID and these models will struggle and fail due to the fundamental statistical assumptions of modern AI techniques. I also agree that as a result, we will normalize the collection of personal data in the name social progress.

Re: How to recognize AI snake oil [pdf]

#146

Post by AareyBaba »

JoeAltmaier wrote: post #105
reggieband wrote: post #58

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…

The San Jose Mercury News had a weather-forecasting contest. It was won one year by a 12-year-old, who's algorithm was "The weather tomorrow will be the same as the weather today". A kind of AI I guess.

Brilliant. I think YouTube has arrived at the same algorithm - it picks the videos I watched yesterday to recommend today.

Re: How to recognize AI snake oil [pdf]

#147

Post by mark-r »

chimi wrote: post #125
formalsystem wrote: post #15

Over the years my heuristic has turned into: "Did the team formulate their problem as a supervised learning problem?" - If not it's probably BS. In longform if anyone is interested https://medium.com/@marksaroufim/can-deep-learning-solve-my-... EDIT: I would consider autoencoders, word2vec, Reinforcement Learning examples of turning a different problem into a supervised learning problem EDIT 2: Social functions like…

It's hard to verbalize this, most of it is "intuition" but I think it boils down to "supervised learning is BS." Humans are smarter than computers. How can a human teach a computer how to do something when the human itself can't teach another human that something? We haven't solved that problem. The snake is eating its tail. You can't teach a human how to do something when the methodology to do that is the student tr…

> You can't teach a human how to do something when the methodology to do that is the student trying something and the teacher saying "Yes" or "No".

Strangely, I recently had to complete a cognitive test that was essentially that process. I was given a series of pages, each of which had a number of shapes and a multiple choice answer. I was told whether I chose the correct answer, then the page was flipped to the next problem. The heuristic for the correct answer was changed at intervals during the test, without any warning from the tester. I'm told I did OK.

Re: How to recognize AI snake oil [pdf]

#148

Post by Fr0styMatt88 »

cycrutchfield wrote: post #103
DenisM wrote: post #96

Earlier quoted context omitted.

Today ML can solve some problems. In the future it might solve some problems with advances in the field. Yet other problems will likely remain unsolved, such as the stock market, or the weather, or predicting /dev/rand "Up Next" problem can easily fall into any of the three buckets.

YouTube's "Up Next" recommendations do (significantly) better than random, therefore "Today ML can solve some problems".

I'm pretty sure it does better than 'next video = random(from all of YouTube)' would, but would it be much better than 'next video = random(videos with the same subject or tags as the one playing now)'?

Re: How to recognize AI snake oil [pdf]

#149

Post by dhairya »

simion314 wrote: post #131
cycrutchfield wrote: post #103

Earlier quoted context omitted.

YouTube's "Up Next" recommendations do (significantly) better than random, therefore "Today ML can solve some problems".

>YouTube's "Up Next" recommendations do (significantly) better than random, therefore "Today ML can solve some problems". IMO YT AI is the opposite of intelligent , it still recommends things I disliked. for some reason this basic rule of not showing something that I explicitly disliked was to hard for it to learn, I am wondering if it is truly an AI behind it or just statistics

It's intentional. Controversy is a strong signal for the youtube algorithm.

Re: How to recognize AI snake oil [pdf]

#150

Post by mike_hock »

madrox wrote: post #140
reggieband wrote: post #58

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…

I had a similar experience at one point. A team put energy into building a recommendation system, and were able to demonstrate that the "Recommended for you" content performed better than all other content editorial. After getting challenged a bit, though, turns anything performs better when put under the header "Recommended for you."

> put energy put energy

And they say English doesn't have reduplication!

Post reply on HN