A relevant Twitter thread: https://twitter.com/NeuroStats/status/1192679554306887681 At the risk of projecting, this has the hallmark of bad experimental design. The best experiments are designed to determine which of many theories better account for what we observe. (When I write "you" or "your" below, I don't mean YOU specifically, but anyone designing the kind of experiment you describe.) One model of gravity says…
This is a very good comment. Is this line of reasoning fleshed out and written up somewhere so I could point people to it? (Also, I would like to think more deeply about its implications) > What models/theories of customer behavior were your experiments designed to distinguish between? My guess is "none" because someone thinking about the problem scientifically would start with a single experiment whose results are m…
Nope, I just typed the above off-the-cuff. I could tweet storm it. Would that be useful?
I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…
Personal recommendation systems all have tradeoffs. It's just the nature of curation as an intangible endeavor. You can love "Scarface", "Heat" and "LA Confidential" but still find "Casino" boring ;)
More on such tradeoffs in a recent case study from DeepMind on Google Play Store app recommendations. Even they acknowledge the same techniques that surface 30% cost efficiencies in data center cooling, may not be completely applicable to "taste"
i m not sure, if you factor in the CO2 footprint of computer manufacture, and the fact that AI needs powerful computers & networking to be delivered. Our body carbon is almost 100% recycleable.
If only the carbon footprint of a human was the body carbon. Modern humans have a very heavy carbon footprint, especially in the US. Think of all the things you do and consume and all the carbon involved all thorough the chain. It's a big number. Computers are extremely efficient compared to that.
People you hire on MTurk don't generally work in the US, and have a small fraction of the carbon footprint of an average American.
We won’t ever have an AI winter like in the 70s again. A lot of ML is already very useful across many domains (computer vision, NLP, advertising, etc). Back then, there was almost no personal computing, almost no internet, smol data, and so on. Stuff you need for ML to be useful and used. So what if some corporate hack calls linear regression “AI”? The results speak for themselves. The ML genie is too profitable to g…
Didn't linear regression used to be called "AI" as recently as a decade ago?
It's still better in many cases than modern ML (especially if you incorporate explainability and efficiency as metrics of "better" next to the predictive power), so I wouldn't object much if a company called it "AI". In fact, if I learned that an "AI" behind some product was just linear regression, I'd trust them more.
Perhaps ML could be applied here to help filter out the barrage of AI snake oil. Funding, anyone?
The key claim of the presentation: “For predicting social outcomes, AI is not substantially better than manual scoring using just a few features” Suggests that only a simple heuristic is needed: if the AI salesman claims their product predicts a social outcome, like a candidate’s job performance or a person’s future criminality, it is snake oil.
And I bet I can beat that with a trivial algorithm: if(marketing copy contains the word 'AI') return 'bullshit'.
WRT. presentation, I didn't read it as claiming that just predicting social outcomes is hard; I read it as saying that perception is the only category where "AI" actually works somewhat (from the "automating judgement" part, I'd argue only spam detection works; I'm yet to hear of any effective ML solution for the rest of the bullet points there).
Since we are sharing anecdotes, I can report it's been 20 years of buying stuff on the internet and the combined billions of ad tracking research dollars spent by Amazon and Google have not yet come up with a better algorithm than to bombard me with ads for the exact same thing I just bought .
This can happen when a retargeting campaign doesn't have a 'burn pixel' or conversion event trigger. It's a common oversight, which can cause a re-targeting program to kick-off unnecessarily (or cause ads that have followed you around to become obvious)
The AI innovation must be that they can figure out which marketers are likely to forget to have a burn pixel because those marketers drive more revenue.
Lots of AI is actually large numbers of humans working on small bits of problems that are beyond our ability to automate. Not infrequently these are passed off on the outside as 'ai' startups. There are some good examples too where the companies that use machine learning properly and to good effect. Interestingly they don't blab about it because it is their edge over the competition and often just knowing that someth…
> if it requires advertising it is probably fake, if it is very quiet and successful it is likely genuine. This is true of almost any product being offered for sale. Good advice.
Agreed, and I personally use the following heuristic in my product evaluation: the heavier the intensity of advertising, the worse the product. After correcting for rough company size (bigger company = bigger baseline advertising budget), I found it to be quite accurate.
I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our c…
Since we are sharing anecdotes, I can report it's been 20 years of buying stuff on the internet and the combined billions of ad tracking research dollars spent by Amazon and Google have not yet come up with a better algorithm than to bombard me with ads for the exact same thing I just bought .
That's because:
* They don't have suppression set up
* They're using a conversion tracking platform that is slow
* They're testing the returns conversion hypothesis: you have expressed concrete intent, you have bought the product. If it has 5% return rate, you probably still want it, and there's a 5% chance they need to be in the mix.
> You can't teach a human how to do something when the methodology to do that is the student trying something and the teacher saying "Yes" or "No". Strangely, I recently had to complete a cognitive test that was essentially that process. I was given a series of pages, each of which had a number of shapes and a multiple choice answer. I was told whether I chose the correct answer, then the page was flipped to the next…
You're touching on the "difficulty" in verbalizing it. I see what you mean, because you did learn that the heuristic was changing with just a yes or no. I said you can't teach that way, but you clearly learned that way, so I wasn't exactly correct, but I'm not practically wrong either still I don't think. I wonder, how would an AI perform on the same test. What is the mathematical minimum number of questions on such…
Doesn’t it depend on what you mean by guarantee? The test can’t get 100% certainty, since theoretically you could be flipping a coin each time and miraculously getting it right, for 1000 times in a row. The chance of that is minuscule (1/2^1000), but it’s nonzero. So we’d have to define a cutoff point for guaranteed. The one used generally in many sciences is 1/20 chance (p = 0.05), so that seems like a plausible one, and with that cutoff, I think you’d need five questions passed in a row (1/2^5 = 1/32). Generally, if you want a chance of p, you need log2(1/p) questions in a row passed correctly. However, that only works if your only options are random guessing and having learned the heuristic. If you sorta know the heuristic (eg. right 2/3 of the time), then you’d get the 5 questions right ~13% ((2/3)^5) of the time, which isn’t inside the p = 0.05 range. So you also need to define a range around your heuristic, like knowing it X of the time. Then you’d need log(1/p)/log(1/X) questions. For example, if you wanted to be the same as the heuristic 19/20 times and you wanted to pass the p = 0.05 threshold, you’d need log(1/0.05)/log(1/(19/20)) ~= 59 questions.