How to recognize AI snake oil [pdf] • Hacker News Board

Re: How to recognize AI snake oil [pdf]

#91

Post by wtvanhest » Tue, Nov 19, 2019, 9:35 PM UTC

bulldoa wrote: post #85

reggieband wrote: post #58

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…

Isn't sparse recommendation for videos kind of solved in netflix prize, where the winner uses SVD to extract signature characteristic and recommend videos base on that?

Probably not. I say this because for many months, i would visit netflix and not want to watch anything. Eventually I cancelled my subscription after many years.

I think I'd rather have a random collection of titles than a recommended list for me.

Re: How to recognize AI snake oil [pdf]

#92

Post by cycrutchfield » Tue, Nov 19, 2019, 9:36 PM UTC

reggieband wrote: post #58

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…

If the team can’t even beat random, then I think that says more about your team (or perhaps your features) than about ML as a whole.

Re: How to recognize AI snake oil [pdf]

#93

Post by nilkn » Tue, Nov 19, 2019, 9:36 PM UTC

reggieband wrote: post #58

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…

I wonder if this could be a case of mismatch between what the recommendations system was designed to do and what the business actually needed it to do. Your team evaluated the models based on live KPIs in an A/B testing environment, but did the recommendations team develop the system specifically with those KPIs in mind? Did they ever have access to adequate information to truly solve the problem your team needed solved? And was the same result observed for other uses of their recommendation systems?

Re: How to recognize AI snake oil [pdf]

#94

Post by sgt101 » Tue, Nov 19, 2019, 9:42 PM UTC

HlessClaudesman wrote: post #79

My brush with AI snake oil: I interviewed at a startup that seemed fishy. They offer a fully AI powered customer service chat as an off the shelf black box to banks. I highly suspect that they were a pseudo AI setup. LinkedIn shows that they are light on developers but very heavy on “trainers”, probably the people who actually handle the customers, mostly young graduates in unrelated fields, who may believe that thei…

I guess the idea would be to get a few contracts, pull down some money from those and then go bust as the costs of the Mechanical Turk become evident?

Re: How to recognize AI snake oil [pdf]

#95

Post by eloff » Tue, Nov 19, 2019, 9:45 PM UTC

buboard wrote: post #82

eloff wrote: post #54

Earlier quoted context omitted.

You mean high carbon, low silicon? Because humans usually have a higher carbon footprint than computers, it takes a lot of computers to match one human. Plus we're made of carbon.

i m not sure, if you factor in the CO2 footprint of computer manufacture, and the fact that AI needs powerful computers & networking to be delivered. Our body carbon is almost 100% recycleable.

If only the carbon footprint of a human was the body carbon.

Modern humans have a very heavy carbon footprint, especially in the US. Think of all the things you do and consume and all the carbon involved all thorough the chain. It's a big number. Computers are extremely efficient compared to that.

Re: How to recognize AI snake oil [pdf]

#96

Post by DenisM » Tue, Nov 19, 2019, 9:47 PM UTC

cycrutchfield wrote: post #92

reggieband wrote: post #58

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…

If the team can’t even beat random, then I think that says more about your team (or perhaps your features) than about ML as a whole.

Today ML can solve some problems. In the future it might solve some problems with advances in the field. Yet other problems will likely remain unsolved, such as the stock market, or the weather, or predicting /dev/rand

"Up Next" problem can easily fall into any of the three buckets.

Re: How to recognize AI snake oil [pdf]

#97

Post by bonoboTP » Tue, Nov 19, 2019, 9:49 PM UTC

cycrutchfield wrote: post #92

reggieband wrote: post #58

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…

If the team can’t even beat random, then I think that says more about your team (or perhaps your features) than about ML as a whole.

Or not enough communication with the team, discussing what the objectives are, providing them with good, enough, relevant data to work with etc. I guess in many cases a data science team is expected to just "do their magic", build the AI and then come back and meanwhile not bother anyone else. In other cases, nobody really cares anyway, they just want the buzzword label to be includable in the brochures.

Re: How to recognize AI snake oil [pdf]

#98

Post by DenisM » Tue, Nov 19, 2019, 9:50 PM UTC

sgt101 wrote: post #94

HlessClaudesman wrote: post #79

My brush with AI snake oil: I interviewed at a startup that seemed fishy. They offer a fully AI powered customer service chat as an off the shelf black box to banks. I highly suspect that they were a pseudo AI setup. LinkedIn shows that they are light on developers but very heavy on “trainers”, probably the people who actually handle the customers, mostly young graduates in unrelated fields, who may believe that thei…

I guess the idea would be to get a few contracts, pull down some money from those and then go bust as the costs of the Mechanical Turk become evident?

> go bust as the costs of the Mechanical Turk become evident

I'm afraid you have misspelled "raise a humongous round form SoftBank". It's an easy typo to make, don't feel bad.

Re: How to recognize AI snake oil [pdf]

#99

Post by buboard » Tue, Nov 19, 2019, 9:52 PM UTC

robotresearcher wrote: post #25

buboard wrote: post #14

Earlier quoted context omitted.

You should emphasize that this is Organic AI. It's low carbon and overall greener.

Each unit uses about 100W continuously and emits about 1kg of CO2 per day before adding impact of supporting infrastructure. These things better be smart, because they are not low-footprint.

We just have to hope our robot overlords will not be overly environmentally conscious

Re: How to recognize AI snake oil [pdf]

#100

Post by mezentius » Tue, Nov 19, 2019, 9:52 PM UTC

reggieband wrote: post #58

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…

>>The best performing variant happened to be random.

Some years ago I heard an anecdote from a developer who had worked on a video game about American football. The gist of it was that they had tested various sophisticated systems for an AI opponent to choose a possible offensive/defensive play, but the one that the players often considered the most "intelligent" was the one that simply made random decisions.

In certain domains, I think, it's quite difficult to beat the perceived performance of an AI system that merely makes committed random decisions (i.e. carried out over time) within a set of reasonable choices. If we don’t understand what an agent is doing, we often assume that there is some devious and subtle purpose behind its actions.