Live data from Hacker News

ML is not that good at predicting consumers' choices

statmodeling.stat.columbia.edu

41–50 of 228 posts

Re: ML is not that good at predicting consumers' choices

#41
post #29

I can personally vouch that Amazon, Twitter, and YouTube all do horrible horrible jobs predicting my taste. And they have got worse over the years, not better

That may make sense of you are not the average consumer. Optimizing for the most common case makes sense. I see that with Google search prediction, it's good but many times it predicts very sensible words for general use but not in the topic that I'm interested.

Re: ML is not that good at predicting consumers' choices

#42
Always interesting to see outsiders writing papers about this, using anecdote and unrelated data (mostly political and real world purchase data in this case) to argue that ML doesn't make useful predictions. Meanwhile I look at randomized controlled trial data showing millions of dollars in revenue uplift directly attributable to ML vs non-ML backed conversion pipelines, offsetting the cost of doing the ML by >10x.

It reminds me a lot of other populist folk-science belief, like vaccine hesitancy. Despite overwhelming data to the contrary, a huge portion of the US population believes that they are somehow better off contracting COVID-19 naturally versus getting the vaccine. I think when effect sizes per individual are small and only build up across large populations, people tend to believe whatever aligns best with their identity.

Re: ML is not that good at predicting consumers' choices

#43

Always interesting to see outsiders writing papers about this, using anecdote and unrelated data (mostly political and real world purchase data in this case) to argue that ML doesn't make useful predictions. Meanwhile I look at randomized controlled trial data showing millions of dollars in revenue uplift directly attributable to ML vs non-ML backed conversion pipelines, offsetting the cost of doing the ML by >10x. I…

I think you may be conflating the topics and goals of adjacent exercises; predicting consumer behavior is not the same thing as optimizing a conversion pipeline.

Re: ML is not that good at predicting consumers' choices

#45
post #29

I can personally vouch that Amazon, Twitter, and YouTube all do horrible horrible jobs predicting my taste. And they have got worse over the years, not better

My favorite experience with Amazon:

I had just preordered novel 9 of The Expanse, and I got an email recommending something else from the same authors: novel 8 of the Expanse. A more sensible recommendation engine might have assumed that someone who preorders part n+1 of a series may already have part n. Not to mention that Amazon should have known that I already had novel 8 on my Kindle.

I guess generating personalized recommendations at scale is still too expensive. We just get recommendations based on what other customers with vaguely similar tastes were interested in.

Re: ML is not that good at predicting consumers' choices

#46
post #29

I can personally vouch that Amazon, Twitter, and YouTube all do horrible horrible jobs predicting my taste. And they have got worse over the years, not better

The one thing I've been consistently impressed with is TikTok. If I compare recommendations on YouTube to what I get on my TikTok FYP, it's like comparing a 5-year-old to a college graduate on a math test.

Literally to the point where YouTube never pulls me down into the rabbit hole anymore, I watch one video because it was linked from somewhere else, then I bounce.

Re: ML is not that good at predicting consumers' choices

#47
post #22

You just bought a washing machine... could I interest you in a washing machine?

I think the reason this happens is that when you start looking for washing machines, you start getting ads for them. Then when you buy nobody tells the ad companies that you just bought a washing machine so they still send you ads because they think you’re still looking. Even if you just went straight to the model site and clicked “buy”.

Re: ML is not that good at predicting consumers' choices

#48

Always interesting to see outsiders writing papers about this, using anecdote and unrelated data (mostly political and real world purchase data in this case) to argue that ML doesn't make useful predictions. Meanwhile I look at randomized controlled trial data showing millions of dollars in revenue uplift directly attributable to ML vs non-ML backed conversion pipelines, offsetting the cost of doing the ML by >10x. I…

>Always interesting to see outsiders writing papers about this

I don't think you know who andrew gelman is. Additionally, that's not the conclusion derived from this study.

Re: ML is not that good at predicting consumers' choices

#49

Always interesting to see outsiders writing papers about this, using anecdote and unrelated data (mostly political and real world purchase data in this case) to argue that ML doesn't make useful predictions. Meanwhile I look at randomized controlled trial data showing millions of dollars in revenue uplift directly attributable to ML vs non-ML backed conversion pipelines, offsetting the cost of doing the ML by >10x. I…

Are you really sure you're not just fooling yourselves with your randomized controlled trials? As Feynman famously said, the easiest person to fool is yourself. And in business even more than science, you might even like the results.

Have you ever put this data up against something similar to the peer review system in academia, where several experts from a competing deparment (or ideally competing company) try to pick your results apart, disprove your hypothesis?

Re: ML is not that good at predicting consumers' choices

#50

Always interesting to see outsiders writing papers about this, using anecdote and unrelated data (mostly political and real world purchase data in this case) to argue that ML doesn't make useful predictions. Meanwhile I look at randomized controlled trial data showing millions of dollars in revenue uplift directly attributable to ML vs non-ML backed conversion pipelines, offsetting the cost of doing the ML by >10x. I…

Amazon does a remarkably good job of predicting what I'll buy and I frequently add to my purchases.
Post reply on HN