Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to recognize AI snake oil [pdf]

cs.princeton.edu

Post reply on HN
Print view View original

301–310 of 364 posts

Re: How to recognize AI snake oil [pdf]

#301

Post by radarsat1 »

Enginerrrd wrote: post #190
stefan_ wrote: post #181

Earlier quoted context omitted.

Since we are sharing anecdotes, I can report it's been 20 years of buying stuff on the internet and the combined billions of ad tracking research dollars spent by Amazon and Google have not yet come up with a better algorithm than to bombard me with ads for the exact same thing I just bought .

Right? Recommenders are almost counter productive for me in most cases. I want a recommender to remain broad, not give me increasingly niche recommendations a la youtube. About the only halfway decent recommender system I've interacted with is the various forms of curated lists on Spotify. They seem to actually take a decent stab at it with related but sufficiently different and interesting content. For all Facebook…

What I've always assumed, but now this thread has me doubting myself, but what I've assumed is that these systems, even though they appear to suck at specifically targeting me, must somehow be pretty good on the average, still netting big profits overall even though they don't seem to live up to the promise of getting me to buy stuff. But if everyone has this impression, maybe it doesn't work? I mean, I assume companies like Amazon and Facebook would be pretty good at least at optimizing the total rewards from using recommendation systems, but maybe you can't tell from your own anecdotal use. I have no idea. I'd love to see an analysis that includes aggregate numbers.

Re: How to recognize AI snake oil [pdf]

#302

Post by n2j3 »

xmprt wrote: post #269
keyle wrote: post #161

Earlier quoted context omitted.

It's quite common in games for AI to pick a random decision. Simply put, a good AI is a character/NPC that appears to have a mind of its own, and its own life. Nothing beats random at explaining someone's behaviour based on a personal history you don't know. If AI responded/acted based on a predefined set of patterns that could be recognizable, the player would automatically feel it (pattern matching) and makes the N…

I'm not sure what the term for it is, but humans have an uncanny ability to ascribe meaning to pure randomness. I'm not surprised a random AI can appear smart.

When your audio player picks these random tracks https://i.imgur.com/QRoGQRy.png (example from earlier in the day) you start believing in a Higher Power..

Re: How to recognize AI snake oil [pdf]

#303

Post by XuMiao »

TeMPOraL wrote: post #178
mike_hock wrote: post #155

Earlier quoted context omitted.

Seriously. Why would I want to watch a video that I've already watched (unless it's music maybe)?

They have a lot of videos. You're statistically unlikely to care about any randomly selected one. By watching a video, you establish that it's interesting enough for you to watch. It's much more likely that you'll want to rewatch it than that you'll want to view a random video. I'm only half-joking here. To me, YT algorithm seems to be a mix of "show random videos you've already watched" + "show random videos from ch…

I prefer a user driven random walk. Like a multi-arm bandit over a hierarchical graph instead of a stuck-in-local-minimun + noise recommender. But no one does it.

Years ago, there is an app stumble upon. I always find it engaging. Hard to get bored.

Re: How to recognize AI snake oil [pdf]

#304

Post by radarsat1 »

jusonchan81 wrote: post #287

I once figured I can use ML to predict stock market movements and I went about buying 5 years of data from CBOE and started working on modeling. At one point I managed to get fairly accurate predictions of movements and I thought I had struck a lottery. Turns out my training data had tomorrow’s movement indicator in it which I accidentally added. Despite that dumb mistake it was only accurate up to 80% or so. That’s…

I went down that path briefly too. After having trouble getting results, I realized I needed to check the fundamentals. What did it in for me was finally just doing a simple correlation analysis of earlier prices to later prices, and realizing there is NO CORRELATION. I realized, if there's nothing to predict, no amount of ML will magically figure it out for me, no matter how complex the indicator.

(I was just looking at single products.. I suppose this may be completely different when considering multiple products and markets and integrating external information like news sources.. not at all saying that ML can't be applied to the stock market, just that it's not nearly as simple as looking at previous prices to predict future prices, as everyone says... the point being, always check basic correlations!)

Re: How to recognize AI snake oil [pdf]

#306

Post by specialist »

JoeAltmaier wrote: post #105
reggieband wrote: post #58

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…

The San Jose Mercury News had a weather-forecasting contest. It was won one year by a 12-year-old, who's algorithm was "The weather tomorrow will be the same as the weather today". A kind of AI I guess.

Just showing a customer's previously viewed items accounted for 85% of the "lift" credited to the recommendations team I worked on.

Re: How to recognize AI snake oil [pdf]

#307

Post by lawlessone »

vishwajeetv wrote: post #277
reggieband wrote: post #58

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…

Interestingly, Steam's recommendation algorithm to show "similar to a game" by their 'learning machines' works very well. I have found really good games via that, and none of them did show up in the regular recommendations/carousel on top.

pretty sure it just uses the tags players give it, strategy, sci-fi , openworld etc.

Re: How to recognize AI snake oil [pdf]

#308

Post by bildung »

chimi wrote: post #156
tensor wrote: post #142

Earlier quoted context omitted.

Supervised learning in machine learning is nothing remotely like a human teaching anyone anything. It's a very clear mathematical formulation of what the objective is and how the algorithm can improve itself against that objective. The closest analogy for humans would be to define a metric and ask a human to figure out how to maximize that metric. That's something we're often pretty good at doing, often in ways that…

> Supervised learning in machine learning is nothing remotely like a human teaching anyone anything. I disagree, I think it's exactly the same. As an example, a human teaching a human how to use an orbital sander to smooth out the rough grain of a piece of wood. The teacher sees the student bearing down really hard with the sander and hears the RPM's of the sander declining as measured by the frequency of the sound.…

Machine learning really almost nothing in common with most types of human learning. The only type of learning that has similarities is associative learning (think Pavlovs dogs studies).

The human learning situation you describe works quite differently, though: The student sees either the device alone or the teacher using the device to demonstrate its functionality. This is the moment most of the actual learning happens: The student creates internal concepts of the device and its interactions with the surroundings. As a result the student can immediately use the decive more or less correctly. What's left is just some finetuning of parameters like movement vectors, movement speed, applied pressure etc.

If the student would work like ML, it would: hold the device in random ways, like on the cord, the disc, the actual grip. After a bunch of right/wrong responses she would settle on using the grip mostly. Then (or in parallel) the student would try out random surfaces to use the device on: the own hand (wrong), the face of the teacher (wrong), the wall (wrong), the wood (right), the table (wrong) etc. After a bunch of retries she would settle on using the device on the wood mostly.

It's easy to overlook the actual cognitive accomplishments of us humans in menial tasks like this one because most of it happens unconsciously. It's not the "I" that is creating the cognitive concepts.

Re: How to recognize AI snake oil [pdf]

#309

Post by KnightOfWords »

streetcat1 wrote: post #235
tlb wrote: post #88

The economic point of AI decision systems is that they can make an automated decision for $0.0001 of computer time, instead of $10 for a few minutes of an expert's time. For something like spam filtering, you obviously need the cheap solution. But you don't need that for the social interventions where you're making a decision about parole or something. You can spend $10 (or $1000) of people's time to make those decis…

Moreover, the big advantage of AI is consistency. I.e. it does not get tired, bored, lose motivation, etc. Hence it might better to get 70% correct answer all of the time, than 90% (human) some of the time.

On hard problems these AIs are doing no better than chance. Nor is a badly trained AI in any way consistent.

Re: How to recognize AI snake oil [pdf]

#310

Post by GordonS »

claytonjy wrote: post #174
HlessClaudesman wrote: post #79

My brush with AI snake oil: I interviewed at a startup that seemed fishy. They offer a fully AI powered customer service chat as an off the shelf black box to banks. I highly suspect that they were a pseudo AI setup. LinkedIn shows that they are light on developers but very heavy on “trainers”, probably the people who actually handle the customers, mostly young graduates in unrelated fields, who may believe that thei…

I have a feeling I know _exactly_ which company you're talking about...so it's either just that obvious, or there's more than one of these, or both!

I was going to say the same thing!

A couple of weeks ago such a startup based in London contacted me on LinkedIn - the product really hyped AI, but it all seemed very dubious. My guess was it was really a mix of a simple chatbot with a Mechanical Turk-style second line.

Post reply on HN