Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to recognize AI snake oil [pdf]

cs.princeton.edu

Post reply on HN
Print view View original

171–180 of 364 posts

Re: How to recognize AI snake oil [pdf]

#171

Post by reggieband »

nilkn wrote: post #93
reggieband wrote: post #58

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…

I wonder if this could be a case of mismatch between what the recommendations system was designed to do and what the business actually needed it to do. Your team evaluated the models based on live KPIs in an A/B testing environment, but did the recommendations team develop the system specifically with those KPIs in mind? Did they ever have access to adequate information to truly solve the problem your team needed sol…

> did the recommendations team develop the system specifically with those KPIs in mind?

Yes they did - in fact they had input on defining them and helped in tracking them.

> Did they ever have access to adequate information to truly solve the problem your team needed solved?

They believed so. Their team was also responsible for our company data warehousing so they knew even better than me what data was available. Basically any piece of data that could be available they had access to.

> And was the same result observed for other uses of their recommendation systems?

I did not have first-hand access to the results of their use in other recommendation contexts. As I mentioned in my original post I only had second-hand accounts from other teams that went the same route. They reported similar results to me.

Re: How to recognize AI snake oil [pdf]

#172

Post by 2sk21 »

mezentius wrote: post #100
reggieband wrote: post #58

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…

>>The best performing variant happened to be random. Some years ago I heard an anecdote from a developer who had worked on a video game about American football. The gist of it was that they had tested various sophisticated systems for an AI opponent to choose a possible offensive/defensive play, but the one that the players often considered the most "intelligent" was the one that simply made random decisions. In cert…

It is well known that tit-for-tat is the best strategy for iterated prisoners' dilemma. See https://en.wikipedia.org/wiki/Tit_for_tat

Re: How to recognize AI snake oil [pdf]

#173

Post by nsainsbury »

It's a bit of a harsh heuristic, but after working now on several projects involving ML/AI and reading and watching about the experience of others in the industry too, I've come to associate most claims of ML as snake oil.

In industry today, I believe very few businesses are reaping much benefit from ML as compared to trivial statistical/analytical tools (linear regression, most popular recommenders, common sense improvements/optimizations, etc.). The only real benefit I would argue ML has brought for businesses has been in marketing to the general lay audience and misleading investors.

The main reason for this in my opinion is you can't really just come in and make recommendations/improvements to a given problem domain without deeply understanding that domain back to front - and that's an understanding that academic types that get hired to build ML systems almost never have. You can't stand at an arms length from real business problems and just throw maths at them and expect to make good (or even sensible) recommendations.

Re: How to recognize AI snake oil [pdf]

#174

Post by claytonjy »

HlessClaudesman wrote: post #79

My brush with AI snake oil: I interviewed at a startup that seemed fishy. They offer a fully AI powered customer service chat as an off the shelf black box to banks. I highly suspect that they were a pseudo AI setup. LinkedIn shows that they are light on developers but very heavy on “trainers”, probably the people who actually handle the customers, mostly young graduates in unrelated fields, who may believe that thei…

I have a feeling I know _exactly_ which company you're talking about...so it's either just that obvious, or there's more than one of these, or both!

Re: How to recognize AI snake oil [pdf]

#175

Post by mike_hock »

simion314 wrote: post #131
cycrutchfield wrote: post #103

Earlier quoted context omitted.

YouTube's "Up Next" recommendations do (significantly) better than random, therefore "Today ML can solve some problems".

>YouTube's "Up Next" recommendations do (significantly) better than random, therefore "Today ML can solve some problems". IMO YT AI is the opposite of intelligent , it still recommends things I disliked. for some reason this basic rule of not showing something that I explicitly disliked was to hard for it to learn, I am wondering if it is truly an AI behind it or just statistics

Isn't AI statistics?

Re: How to recognize AI snake oil [pdf]

#176

Post by chimi »

mark-r wrote: post #144
wilg wrote: post #137

Earlier quoted context omitted.

Well... why is it necessary that we can teach a human to do something in order to teach a machine to do it?

Teaching a human is a heuristic for understanding the problem well enough to teach a machine.

I agree and rather than post a sibling response, I'll add that I think it's necessary today, simply because we don't have AGI, yet. And also point out that we are talking about determining if AI is snake oil or not. There may be some scenarios where we can teach a computer to do something we can't teach a human to do, I can't think of any off the top of my head, but if we can't, then I'm going to be super doubtful that an AI software can do it better than a human, if at all.

AGI, in the singularity sense, will be solving problems before we even identify them as problems. Experts in a field can do this for the layman already and I think it's possible. Some don't. I do.

It'll be super interesting when it flips! When the student becomes the master and we, as a species, start learning from the computer. You can kind of get a sense of this from the Deep Mind founder's presentation on their AI learning how to play the old Atari game Breakout. He says when their engineers watched the computer play the game, it had developed techniques the engineers who wrote the program hadn't even thought of.

Even still, the engineers could teach another human how to play Breakout, so yes, I do believe they did in fact create a software to play Breakout better than they could.

Re: How to recognize AI snake oil [pdf]

#177

Post by james_s_tayler »

atomicity wrote: post #163
JimmyRuska wrote: post #160

Earlier quoted context omitted.

Well to be fair that's how all employers also hire. If you did a good job at the last company you'll probably do a good job here. If you did a good job yesterday, you'll probably do a good job today. For the most part they are usually correct.

I hope you are joking since the industry collectively knows how much employers/interviewers value algorithms-based coding interview, which doesn't correlate strongly with performance. Even if you are talking about senior positions where they don't matter, then you should know that people hire someone they know+like who did decently well, rather than the truly best on the market.

That's SF/Big Tech. The rest of the world basically works like the stated algorithm.

Re: How to recognize AI snake oil [pdf]

#178

Post by TeMPOraL »

mike_hock wrote: post #155
AareyBaba wrote: post #146

Earlier quoted context omitted.

Brilliant. I think YouTube has arrived at the same algorithm - it picks the videos I watched yesterday to recommend today.

Seriously. Why would I want to watch a video that I've already watched (unless it's music maybe)?

They have a lot of videos. You're statistically unlikely to care about any randomly selected one. By watching a video, you establish that it's interesting enough for you to watch. It's much more likely that you'll want to rewatch it than that you'll want to view a random video.

I'm only half-joking here. To me, YT algorithm seems to be a mix of "show random videos you've already watched" + "show random videos from channels and users you watched" + "show the most popular videos in last few hours/days/weeks". It's pretty much worthless, but what are we expecting? Like all things ad economy, the primary metric here isn't whether you like the recommended content, or whether that content challenges you to grow - it's maximizing the amount of videos you watch, because videos are the vehicles for delivering ads to you.

Re: How to recognize AI snake oil [pdf]

#179

Post by scabarott »

MadWombat wrote: post #136
Hitton wrote: post #109

Earlier quoted context omitted.

That doesn't work. Everything is called AI these days and in mountains of bullshit there are also some actually useful results, these few are not snake oil.

Somewhere out there, a biotech R&D company has developed an effective penis enlargement treatment. Unfortunately they have been having some trouble reaching potential customers.

This is a tragedy for half of humanity. Actually, all humanity come to think of it

Re: How to recognize AI snake oil [pdf]

#180

Post by Nition »

HlessClaudesman wrote: post #79

My brush with AI snake oil: I interviewed at a startup that seemed fishy. They offer a fully AI powered customer service chat as an off the shelf black box to banks. I highly suspect that they were a pseudo AI setup. LinkedIn shows that they are light on developers but very heavy on “trainers”, probably the people who actually handle the customers, mostly young graduates in unrelated fields, who may believe that thei…

Sounds a lot like Zach:

Pt 1: https://thespinoff.co.nz/the-best-of/06-03-2018/the-mystery-...

Pt 2: https://thespinoff.co.nz/the-best-of/09-03-2018/the-mystery-...

Post reply on HN