How to recognize AI snake oil. The salesman calls it AI and claims it is not simply statistical inference.
How to recognize AI snake oil [pdf]
231–240 of 364 posts
Re: How to recognize AI snake oil [pdf]
#232I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…
Since we are sharing anecdotes, I can report it's been 20 years of buying stuff on the internet and the combined billions of ad tracking research dollars spent by Amazon and Google have not yet come up with a better algorithm than to bombard me with ads for the exact same thing I just bought .
Sometimes it also recommends other brands of the thing I just bought. Like a cell phone: I didn't buy Samsung, but now I see their ads. I guess that is a very very small improvement- Maybe I'm the type of consumer who gets a new phone every week?
Re: How to recognize AI snake oil [pdf]
#233Earlier quoted context omitted.
>>The best performing variant happened to be random. Some years ago I heard an anecdote from a developer who had worked on a video game about American football. The gist of it was that they had tested various sophisticated systems for an AI opponent to choose a possible offensive/defensive play, but the one that the players often considered the most "intelligent" was the one that simply made random decisions. In cert…
It's quite common in games for AI to pick a random decision. Simply put, a good AI is a character/NPC that appears to have a mind of its own, and its own life. Nothing beats random at explaining someone's behaviour based on a personal history you don't know. If AI responded/acted based on a predefined set of patterns that could be recognizable, the player would automatically feel it (pattern matching) and makes the N…
Unless you consider that randomness, whatever source, is some form of intelligence. (I am only half-joking here).
Re: How to recognize AI snake oil [pdf]
#234My company is sourcing AI from MTurk. It's actually cheaper than running fat GPU model training instances. The network learns fast and adapts well to changes in inputs. I envision the sticker "human inside" strapped on our algorithms.
What about data privacy? do your customers knows that random Turks look at their data?
Re: How to recognize AI snake oil [pdf]
#235The economic point of AI decision systems is that they can make an automated decision for $0.0001 of computer time, instead of $10 for a few minutes of an expert's time. For something like spam filtering, you obviously need the cheap solution. But you don't need that for the social interventions where you're making a decision about parole or something. You can spend $10 (or $1000) of people's time to make those decis…
Moreover, the big advantage of AI is consistency. I.e. it does not get tired, bored, lose motivation, etc.
Hence it might better to get 70% correct answer all of the time, than 90% (human) some of the time.
Re: How to recognize AI snake oil [pdf]
#236The author says "AI is already at or beyond human accuracy in all the tasks on this slide and is continuing to get better rapidly" and one of his examples is "Medical diagnosis from scans". That is an example of precisely the sort of snake oil hype he's berating in the social prediction category. In an extremely narrow sense of pattern recognition of some "image features", i.e. 5% of what a radiologist actually does,…
Author here. I appreciate your criticism. What I had in mind was more along the lines of Google's claims around diabetic retinopathy. I received feedback very similar to yours, i.e. that those claims are based on an extremely narrow problem formulation: https://twitter.com/MaxALittle/status/1196957870853627904
I will correct this in future versions of the talk and paper.
Re: How to recognize AI snake oil [pdf]
#237The author says "AI is already at or beyond human accuracy in all the tasks on this slide and is continuing to get better rapidly" and one of his examples is "Medical diagnosis from scans". That is an example of precisely the sort of snake oil hype he's berating in the social prediction category. In an extremely narrow sense of pattern recognition of some "image features", i.e. 5% of what a radiologist actually does,…
FYI, I too looked at his categorisation of what was/was advancing/was not snake oil and didn't exactly agree with all his decisions either.
Medical imaging diagnosis was one of them.
Speech recognition/transcription was another. I don't know if it's my accent or my speech patterns(though foreigners regularly compliment my wife and myself on our pronunciation), but the tech hasn't gotten noticeably better for me since the days of Dragon natural speaking, and that was, what...10 years ago?
Sure, I can "hey Google/siri/alexa" a handful of predefined commands, but I still have to talk in a sort of stoccato "I am talking to a computer" voice, it still only gets it right 90% of the time, and God help you if you try anything new/natural not in the form of "writing Boolean logic programs with my voice".
Re: How to recognize AI snake oil [pdf]
#238Earlier quoted context omitted.
Since we are sharing anecdotes, I can report it's been 20 years of buying stuff on the internet and the combined billions of ad tracking research dollars spent by Amazon and Google have not yet come up with a better algorithm than to bombard me with ads for the exact same thing I just bought .
This can happen when a retargeting campaign doesn't have a 'burn pixel' or conversion event trigger. It's a common oversight, which can cause a re-targeting program to kick-off unnecessarily (or cause ads that have followed you around to become obvious)
And yet I haven't seen one ad that has a 'not interested' feature similar to YouTube.
I mean, if I could stop seeing washing machines or whatever, I'd probably click it.
Re: How to recognize AI snake oil [pdf]
#239Earlier quoted context omitted.
Since we are sharing anecdotes, I can report it's been 20 years of buying stuff on the internet and the combined billions of ad tracking research dollars spent by Amazon and Google have not yet come up with a better algorithm than to bombard me with ads for the exact same thing I just bought .
I just spent 15m on Amazon trying to prod the recommendation algorithm into finding something I actually wanted to buy so I could get above the "free delivery threshold". Think about that. I wanted to spend money. I wasn't too fussy what it was. Amazon has a decade of my purchasing and browsing history. And they still failed.
Ha. Last time I wanted to buy something on Amazon, their search page kept freezing every time I load it: 100% CPU load. Because I really wanted to buy that thing, I spent 30 mins debugging their silly scripts and found one for-loop that tries to find a non-existent element. Unfortunately, I couldn't figure out a way to enable my fix in a minified script, as reloading the page kept loading the original script.
Re: How to recognize AI snake oil [pdf]
#240I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…
This reminds me of a job interview I was on. I was asked about how I would use AI/machine learning for their problem space. Since they seemed to be smart and level-headed, I answered honestly, "Pick something unimportant, use a machine learning algorithm just to get familiar with the tools, ignore the result unless it happens to work, then put machine learning in your marketing materials. But keep track of it, and if…
Speaking of jobs and interviews. I am yet to find a job board which does not show JavaScript jobs when searching for Java jobs. Some of them claim to use AI. :)