Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to recognize AI snake oil [pdf]

cs.princeton.edu

Post reply on HN
Print view View original

221–230 of 364 posts

Re: How to recognize AI snake oil [pdf]

#221

Post by TrackerFF »

dekhn wrote: post #20

Top textual feature predicting snake oil: calling the product AI rather than ML.

Sometimes you just need to do it that way, because the people buying do not know what machine learning is - even though they've heard about AI.

For example, I was networking for some jobs in data science - and was approached by some energy company. Struck up a conversation with the guy (older exec), and he said "so I hear you have a background from AI, correct?" to which I replied "I have a degree in Machine Learning, and have worked with etc." - he just replied "Oh, we don't really need any mechanical engineers now"

So I asked him, mechanical engineering? He pointed out that I said MACHINE, and assumed it had something to do with, you know, physical machines and stuff - so in the domain of mechanical engineering.

So, from then, I went easy on using Machine Learning unless I was fairly confident the other guy had some domain knowledge. If I talk with non-technical salespeople, or older executives, I just leave it at AI.

Re: How to recognize AI snake oil [pdf]

#222

Post by bodeadly »

Isn't "AI" pretty much snake oil. IIRC Artificial Intelligence used to mean a computer that could think like a person. But that just is not the case. Even with IBM ads with a computer talking to people saying it's going to fix the network and stop cyber attacks that is just complete nonsense. And it will probably always be nonsense because of course there's no way a computer can think like a person because a computer is not a person. It did not grow up and fall of it's bike and skin it's knee and take a road trip to the rock concert and meet someone and so on. AI is being used as a marketing term to compensate for the fact that sophisticated pattern recognition algorithms and the like are not particularly marketable even if they are useful.

Re: How to recognize AI snake oil [pdf]

#223

Post by [deleted] »

JoeAltmaier wrote: post #105
reggieband wrote: post #58

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…

The San Jose Mercury News had a weather-forecasting contest. It was won one year by a 12-year-old, who's algorithm was "The weather tomorrow will be the same as the weather today". A kind of AI I guess.

[deleted]

Re: How to recognize AI snake oil [pdf]

#224

Post by kspacewalk2 »

The author says "AI is already at or beyond human accuracy in all the tasks on this slide and is continuing to get better rapidly" and one of his examples is "Medical diagnosis from scans". That is an example of precisely the sort of snake oil hype he's berating in the social prediction category.

In an extremely narrow sense of pattern recognition of some "image features", i.e. 5% of what a radiologist actually does, he's probably right. But context is the other 95%, and AI is nowhere close to being able to approach expert accuracy in that. It's a goal as far away from reality as AGI.

"AI" tools will probably improve the productivity of radiologists, and there are statistical learning tools that already kind of do that (usually not actually widely used in medical practice, you can say yet, I can say who knows but nice prototype). But actual diagnosis, like the part where an MD makes a judgement call and the part which malpractice insurance is for? Not in any of our lifetimes.

A radiologist friend complains that it's been 10+ years since they've been using speech recognition instead of a human transcriptionist, and all the systems out there are still really bad. Recognizing medical lingo is something you can probably achieve with more training data, but the software that sometimes drops "not" from a scan report is a cost-cutting measure, not a productivity tool. It makes the radiologist worse off because he's got to waste his time proofreading the hell out of it, but the hospital saves money.

Re: How to recognize AI snake oil [pdf]

#225

Post by sanchezdev »

fspeech wrote: post #206
sanchezdev wrote: post #132

Earlier quoted context omitted.

I didn't mean to make it sound incidental although I do see your point. Just wanted to chime in with how important having a labeled dataset is for a successful ML project.

I think the point is labeling itself is very difficult except for special and limited domains. Manually constructed labels, like feature engineering, are not robust and do not advance the field in general.

That makes sense. I'm coming from the angle of applied ML where solutions need to solve a business problem rather than advance the field of ML. In consulting many problems can't be solved well without a labeled dataset and in lieu of one, less credible data scientists will claim they can solve it in an unsupervised manner.

Re: How to recognize AI snake oil [pdf]

#226

Post by munin »

chimi wrote: post #165
mark-r wrote: post #147

Earlier quoted context omitted.

> You can't teach a human how to do something when the methodology to do that is the student trying something and the teacher saying "Yes" or "No". Strangely, I recently had to complete a cognitive test that was essentially that process. I was given a series of pages, each of which had a number of shapes and a multiple choice answer. I was told whether I chose the correct answer, then the page was flipped to the next…

You're touching on the "difficulty" in verbalizing it. I see what you mean, because you did learn that the heuristic was changing with just a yes or no. I said you can't teach that way, but you clearly learned that way, so I wasn't exactly correct, but I'm not practically wrong either still I don't think. I wonder, how would an AI perform on the same test. What is the mathematical minimum number of questions on such…

Sounds like Ravens progressive matricies.

Re: How to recognize AI snake oil [pdf]

#227

Post by nradov »

Enginerrrd wrote: post #190
stefan_ wrote: post #181

Earlier quoted context omitted.

Since we are sharing anecdotes, I can report it's been 20 years of buying stuff on the internet and the combined billions of ad tracking research dollars spent by Amazon and Google have not yet come up with a better algorithm than to bombard me with ads for the exact same thing I just bought .

Right? Recommenders are almost counter productive for me in most cases. I want a recommender to remain broad, not give me increasingly niche recommendations a la youtube. About the only halfway decent recommender system I've interacted with is the various forms of curated lists on Spotify. They seem to actually take a decent stab at it with related but sufficiently different and interesting content. For all Facebook…

Likewise Pandora does a decent job of picking songs for me based on what I previously liked. It's not perfect but far better than random.

Re: How to recognize AI snake oil [pdf]

#228

Post by mobjack »

reggieband wrote: post #171
nilkn wrote: post #93

Earlier quoted context omitted.

I wonder if this could be a case of mismatch between what the recommendations system was designed to do and what the business actually needed it to do. Your team evaluated the models based on live KPIs in an A/B testing environment, but did the recommendations team develop the system specifically with those KPIs in mind? Did they ever have access to adequate information to truly solve the problem your team needed sol…

> did the recommendations team develop the system specifically with those KPIs in mind? Yes they did - in fact they had input on defining them and helped in tracking them. > Did they ever have access to adequate information to truly solve the problem your team needed solved? They believed so. Their team was also responsible for our company data warehousing so they knew even better than me what data was available. Bas…

Some ideas seem to attract smart people like moths to a flame.

It seems like everyone who joins my company to shake things up follows the same path of wanting personalized content to acquire new customers.

But in reality we just don't have enough data points on people before they become customers to segment people that way. Even if we could, being able to accurately

Every time I see people go through the motions of attempting to implement this until they eventually give up.

This idea looks like an obvious win and big companies have done them before with success, but is extremely hard to impossible to pull off for our small company.

Re: How to recognize AI snake oil [pdf]

#229

Post by democracy »

stefan_ wrote: post #181
reggieband wrote: post #58

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…

Since we are sharing anecdotes, I can report it's been 20 years of buying stuff on the internet and the combined billions of ad tracking research dollars spent by Amazon and Google have not yet come up with a better algorithm than to bombard me with ads for the exact same thing I just bought .

The marketing budgets are there to be spent :)

Re: How to recognize AI snake oil [pdf]

#230

Post by mathattack »

stefan_ wrote: post #181
reggieband wrote: post #58

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…

Since we are sharing anecdotes, I can report it's been 20 years of buying stuff on the internet and the combined billions of ad tracking research dollars spent by Amazon and Google have not yet come up with a better algorithm than to bombard me with ads for the exact same thing I just bought .

Yes - just in case you want two or stopped the purchase along the way.
Post reply on HN