Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to recognize AI snake oil [pdf]

cs.princeton.edu

Post reply on HN
Print view View original

181–190 of 364 posts

Re: How to recognize AI snake oil [pdf]

#181

Post by stefan_ »

reggieband wrote: post #58

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…

Since we are sharing anecdotes, I can report it's been 20 years of buying stuff on the internet and the combined billions of ad tracking research dollars spent by Amazon and Google have not yet come up with a better algorithm than to bombard me with ads for the exact same thing I just bought.

Re: How to recognize AI snake oil [pdf]

#182

Post by dgb23 »

2sk21 wrote: post #172
mezentius wrote: post #100

Earlier quoted context omitted.

>>The best performing variant happened to be random. Some years ago I heard an anecdote from a developer who had worked on a video game about American football. The gist of it was that they had tested various sophisticated systems for an AI opponent to choose a possible offensive/defensive play, but the one that the players often considered the most "intelligent" was the one that simply made random decisions. In cert…

It is well known that tit-for-tat is the best strategy for iterated prisoners' dilemma. See https://en.wikipedia.org/wiki/Tit_for_tat

It may sound similar but offensive vs. defensive behavior in (video) game strategy is a much different concept than cooperation in game theory.

Re: How to recognize AI snake oil [pdf]

#183

Post by claytonjy »

BlueTemplar wrote: post #130
rm_-rf_slash wrote: post #37

Earlier quoted context omitted.

We won’t ever have an AI winter like in the 70s again. A lot of ML is already very useful across many domains (computer vision, NLP, advertising, etc). Back then, there was almost no personal computing, almost no internet, smol data, and so on. Stuff you need for ML to be useful and used. So what if some corporate hack calls linear regression “AI”? The results speak for themselves. The ML genie is too profitable to g…

Didn't linear regression used to be called "AI" as recently as a decade ago?

it still is, but the people who mean "regression" when they say "AI" will generally not admit this

Re: How to recognize AI snake oil [pdf]

#184

Post by onlyrealcuzzo »

mrweasel wrote: post #27
avip wrote: post #10

My company is sourcing AI from MTurk. It's actually cheaper than running fat GPU model training instances. The network learns fast and adapts well to changes in inputs. I envision the sticker "human inside" strapped on our algorithms.

Even companies like Facebook, Apple and Google employee humans to do work that people believe is done by "computers" and non of the companies seem keen on informing the public that they do in fact have humans scanning through massive amounts of data. So perhaps it is in fact cheaper, or the problems they face remains to hard for current types of AI. Given the number of people Facebook employees to censor content and…

I'm aware of moderation. What else?

Re: How to recognize AI snake oil [pdf]

#185

Post by propogandist »

stefan_ wrote: post #181
reggieband wrote: post #58

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…

Since we are sharing anecdotes, I can report it's been 20 years of buying stuff on the internet and the combined billions of ad tracking research dollars spent by Amazon and Google have not yet come up with a better algorithm than to bombard me with ads for the exact same thing I just bought .

This can happen when a retargeting campaign doesn't have a 'burn pixel' or conversion event trigger. It's a common oversight, which can cause a re-targeting program to kick-off unnecessarily (or cause ads that have followed you around to become obvious)

Re: How to recognize AI snake oil [pdf]

#186

Post by throwaway2048 »

mike_hock wrote: post #155
AareyBaba wrote: post #146

Earlier quoted context omitted.

Brilliant. I think YouTube has arrived at the same algorithm - it picks the videos I watched yesterday to recommend today.

Seriously. Why would I want to watch a video that I've already watched (unless it's music maybe)?

Im guessing things like 5 year olds watching the same video 300 times randomly on their parent's account has irreversibly tainted the Googletron into thinking people like to watch the same video over and over again.

Hundreds of times I have told youtube im not interested in a recommended video that i have already watched, seems to be completely ignored.

Re: How to recognize AI snake oil [pdf]

#187

Post by Thrymr »

chimi wrote: post #156
tensor wrote: post #142

Earlier quoted context omitted.

Supervised learning in machine learning is nothing remotely like a human teaching anyone anything. It's a very clear mathematical formulation of what the objective is and how the algorithm can improve itself against that objective. The closest analogy for humans would be to define a metric and ask a human to figure out how to maximize that metric. That's something we're often pretty good at doing, often in ways that…

> Supervised learning in machine learning is nothing remotely like a human teaching anyone anything. I disagree, I think it's exactly the same. As an example, a human teaching a human how to use an orbital sander to smooth out the rough grain of a piece of wood. The teacher sees the student bearing down really hard with the sander and hears the RPM's of the sander declining as measured by the frequency of the sound.…

But that's not at all how "supervised learning" works. You would do something like have a thousand sanded pieces of wood and columns of attributes of the sanding parameters that were used, and have a human label the wood pieces that meet the spec. Then you solve for the parameters that were likely to generate those acceptable results. ML is brute force compared with the heuristics that human learning can apply. And ML never* gives you results that can be generalized with simple rules.

* excepting some classes of expert systems

Re: How to recognize AI snake oil [pdf]

#188

Post by mezentius »

keyle wrote: post #161
mezentius wrote: post #100

Earlier quoted context omitted.

>>The best performing variant happened to be random. Some years ago I heard an anecdote from a developer who had worked on a video game about American football. The gist of it was that they had tested various sophisticated systems for an AI opponent to choose a possible offensive/defensive play, but the one that the players often considered the most "intelligent" was the one that simply made random decisions. In cert…

It's quite common in games for AI to pick a random decision. Simply put, a good AI is a character/NPC that appears to have a mind of its own, and its own life. Nothing beats random at explaining someone's behaviour based on a personal history you don't know. If AI responded/acted based on a predefined set of patterns that could be recognizable, the player would automatically feel it (pattern matching) and makes the N…

Right-- but often an NPC is merely scenery or traffic, in the sense that their behavior does not compete with your interests. What I found interesting about the football example is that the random strategy of the NPC-oach both suggested a deeper intelligence AND proved to be an apparently effective opponent.

Beyond reinforcing our tendency to project, as you say, a personal history on random behavior, it also highlights what a few other people have commented: that in many non-cooperative situations a committed random strategy is extremely effective, and perhaps more effective than a biased, seemingly "rational" strategy. (For another example, I believe Henrich's "The Secret Of Our Success" discusses the possible adaptive benefits of divination as a generator for random strategies among early societies.)

Re: How to recognize AI snake oil [pdf]

#189

Post by TeMPOraL »

claytonjy wrote: post #174
HlessClaudesman wrote: post #79

My brush with AI snake oil: I interviewed at a startup that seemed fishy. They offer a fully AI powered customer service chat as an off the shelf black box to banks. I highly suspect that they were a pseudo AI setup. LinkedIn shows that they are light on developers but very heavy on “trainers”, probably the people who actually handle the customers, mostly young graduates in unrelated fields, who may believe that thei…

I have a feeling I know _exactly_ which company you're talking about...so it's either just that obvious, or there's more than one of these, or both!

It seems to be the latter. In fact, the trick (should I say, fraud?) is so common that they were even several articles about it in the press over the past two or three years. Even the famous x.ai had (and I guess still has) humans doing the work.

https://www.bloomberg.com/news/articles/2016-04-18/the-human...

Re: How to recognize AI snake oil [pdf]

#190

Post by Enginerrrd »

stefan_ wrote: post #181
reggieband wrote: post #58

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…

Since we are sharing anecdotes, I can report it's been 20 years of buying stuff on the internet and the combined billions of ad tracking research dollars spent by Amazon and Google have not yet come up with a better algorithm than to bombard me with ads for the exact same thing I just bought .

Right? Recommenders are almost counter productive for me in most cases. I want a recommender to remain broad, not give me increasingly niche recommendations a la youtube. About the only halfway decent recommender system I've interacted with is the various forms of curated lists on Spotify. They seem to actually take a decent stab at it with related but sufficiently different and interesting content.

For all Facebook knows about me they've always been exceptionally bad at advertising to me, which is remarkable considering what they've got. Google is only very marginally better. Actually, now that I think about it, Amazon's 'customer's also bought' is also pretty bad at the recommendation itself since it not uncommonly recommends incompatible things! ...but it does often succeed at getting me to think more about what else I might need and sometimes leads me to buying other things. At least it's not always recommending the same thing, but rather related things, which is probably a much better way to advertise.

Post reply on HN