Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to recognize AI snake oil [pdf]

cs.princeton.edu

Post reply on HN
Print view View original

201–210 of 364 posts

Re: How to recognize AI snake oil [pdf]

#201

Post by roywiggins »

mezentius wrote: post #188
keyle wrote: post #161

Earlier quoted context omitted.

It's quite common in games for AI to pick a random decision. Simply put, a good AI is a character/NPC that appears to have a mind of its own, and its own life. Nothing beats random at explaining someone's behaviour based on a personal history you don't know. If AI responded/acted based on a predefined set of patterns that could be recognizable, the player would automatically feel it (pattern matching) and makes the N…

Right-- but often an NPC is merely scenery or traffic, in the sense that their behavior does not compete with your interests. What I found interesting about the football example is that the random strategy of the NPC-oach both suggested a deeper intelligence AND proved to be an apparently effective opponent. Beyond reinforcing our tendency to project, as you say, a personal history on random behavior, it also highlig…

It's how in Rock Paper Scissors you can probably do better against a smart opponent by playing randomly than by trying to trick them. At least they can't get in your head, because there's nothing there. You won't do better than chance, but at least you can't do much worse.

Re: How to recognize AI snake oil [pdf]

#202

Post by andybak »

stefan_ wrote: post #181
reggieband wrote: post #58

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…

Since we are sharing anecdotes, I can report it's been 20 years of buying stuff on the internet and the combined billions of ad tracking research dollars spent by Amazon and Google have not yet come up with a better algorithm than to bombard me with ads for the exact same thing I just bought .

I just spent 15m on Amazon trying to prod the recommendation algorithm into finding something I actually wanted to buy so I could get above the "free delivery threshold".

Think about that. I wanted to spend money. I wasn't too fussy what it was. Amazon has a decade of my purchasing and browsing history.

And they still failed.

Re: How to recognize AI snake oil [pdf]

#203

Post by Bostonian »

JoeAltmaier wrote: post #105
reggieband wrote: post #58

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…

The San Jose Mercury News had a weather-forecasting contest. It was won one year by a 12-year-old, who's algorithm was "The weather tomorrow will be the same as the weather today". A kind of AI I guess.

AR(1) models are commonly employed to model time series, and the "same-as-yesterday" model is the case where the AR1 coefficient equals 1. There is is some mean reversion in seasonally-adjusted temperature, so an AR1 coefficient less than 1 should work better.

Re: How to recognize AI snake oil [pdf]

#204

Post by fvncc »

BlueTemplar wrote: post #114
SkyBelow wrote: post #38

What I dislike far more than the idea of using such systems to predict social outcome is that the usage of such systems is done behind closed doors. I would be much more willing to accept such systems if the law required any system to be fully accessible online, including the current neural network, how it was trained, and training data used to train it (if the training data cannot be shared online, then the neural n…

In my country, a relatively recent law added an obligation for the government to give on request a detailed and joe-six-pack-undersandable explanation for how an "algorithm" has reached a decision pertaining to that person. I've therefore started stockpiling popcorn since this law was announced for the inevitable clusterfuck that was going to happen when this law would have to apply to a decision taken using machine…

Maybe they can use something like this?

https://blog.acolyer.org/2019/11/01/optimized-risk-scores/

They optimise a simple set of decision rules which has reasonable accuracy in their application, quite cool really

Re: How to recognize AI snake oil [pdf]

#205

Post by NoodleIncident »

mike_hock wrote: post #155
AareyBaba wrote: post #146

Earlier quoted context omitted.

Brilliant. I think YouTube has arrived at the same algorithm - it picks the videos I watched yesterday to recommend today.

Seriously. Why would I want to watch a video that I've already watched (unless it's music maybe)?

Music is probably the main driver, but I've definitely clicked some recommended videos from a channel I'm subscribed to with infrequent long uploads.

Re: How to recognize AI snake oil [pdf]

#206

Post by fspeech »

sanchezdev wrote: post #132
MiroF wrote: post #84

Earlier quoted context omitted.

As I said, I'm an NLP researcher and practitioner, so you don't need to quote this at me. The unsupervised aspect is the engine driving all modern NLP advancements. Your comment suggests that it is incidental, which is far from the case. Yes, it is often ultimately then used for a downstream supervised task, but it wouldn't work at all without unsupervised training. Indeed, one of the biggest applications of deep NLP…

I didn't mean to make it sound incidental although I do see your point. Just wanted to chime in with how important having a labeled dataset is for a successful ML project.

I think the point is labeling itself is very difficult except for special and limited domains. Manually constructed labels, like feature engineering, are not robust and do not advance the field in general.

Re: How to recognize AI snake oil [pdf]

#207

Post by Bostonian »

SkyBelow wrote: post #38

What I dislike far more than the idea of using such systems to predict social outcome is that the usage of such systems is done behind closed doors. I would be much more willing to accept such systems if the law required any system to be fully accessible online, including the current neural network, how it was trained, and training data used to train it (if the training data cannot be shared online, then the neural n…

A judge using his experience and judgement to subjectively set a jail sentence is as opaque as a proprietary algorithm. He or she may cite reasons for the sentence, but nobody is verifying that judges' sentences are consistent with the criteria they cite.

Re: How to recognize AI snake oil [pdf]

#208

Post by Bostonian »

erikig wrote: post #158
sgt101 wrote: post #87

I read "Why are HR departments apparently so gullible?" and as someone who has worked in a corporate for 20 years I spotted my underwear. The identification of facial recognition as problematic because of accuracy doesn't match my thinking. I believe that the key issue is that given a set of targets facial recognition systems will find near misses from the wider population of all faces offered as candidates, that the…

...I spotted my underwear? What does this mean?

spotted = soiled

Re: How to recognize AI snake oil [pdf]

#209

Post by mc3 »

btilly wrote: post #129
reggieband wrote: post #58

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…

This reminds me of a job interview I was on. I was asked about how I would use AI/machine learning for their problem space. Since they seemed to be smart and level-headed, I answered honestly, "Pick something unimportant, use a machine learning algorithm just to get familiar with the tools, ignore the result unless it happens to work, then put machine learning in your marketing materials. But keep track of it, and if…

That sounds odd, like they don't really need machine learning (unless it is to snare investors?).

Re: How to recognize AI snake oil [pdf]

#210

Post by scottlocklin »

propogandist wrote: post #185
stefan_ wrote: post #181

Earlier quoted context omitted.

Since we are sharing anecdotes, I can report it's been 20 years of buying stuff on the internet and the combined billions of ad tracking research dollars spent by Amazon and Google have not yet come up with a better algorithm than to bombard me with ads for the exact same thing I just bought .

This can happen when a retargeting campaign doesn't have a 'burn pixel' or conversion event trigger. It's a common oversight, which can cause a re-targeting program to kick-off unnecessarily (or cause ads that have followed you around to become obvious)

You'd think with all the "AI" out there, they could match a sales DB entry with the CRM DB entry, but, in fact, they basically can't.
Post reply on HN