Live data from Hacker News

ML is not that good at predicting consumers' choices

statmodeling.stat.columbia.edu

61–70 of 228 posts

Re: ML is not that good at predicting consumers' choices

#61
post #50

Earlier quoted context omitted.

Amazon does a remarkably good job of predicting what I'll buy and I frequently add to my purchases.

Are you the mythical person buying 15 vacuum cleaners at the same time?

They are not at the same time. There are entire days of interval!

Re: ML is not that good at predicting consumers' choices

#62
post #3

While not exactly aligned to the research, I've been surprised how poor Nest Thermostat's learning feature is. The main selling point for Nest is having a "learning thermostat". Perhaps my schedule is just not predictable enough, but the auto-generated temperature schedules it generates after its "learning" period is not even close to what I would manually set up on a normal thermostat. Maybe I'm just an "edge case"…

"Why am I sweating right now? Oh, the Nest set the temperature too high again!" And then after a few instances, I just turn off all the automation and set up a schedule like normal. Same with the "away from home" which seems to randomly think I'm away and I have no idea why. Oh, and the app doesn't show me filter reminders, only the actual device, which I never touch all the way downstairs. There's not even any statu…

>And then after a few instances, I just turn off all the automation and set up a schedule like normal.

If you have a fairly regular life I would think a schedule would outdo ML pretty much all the time, because you know exactly what that schedule should be. ML might be useful for a secret agent whose life is so erratic that a schedule would be useless.

That is to say ML is maybe better than falling back to nothing.

Re: ML is not that good at predicting consumers' choices

#63
post #33

It is the same on Netflix. I have phases where I watch a certain genre for a few weeks and then move on. For example after a few Scandi crime series it is time for something else. However, at the same time my daughter loves Animé and pretty only watch that. It is really hard for an ML algorithm to grab these nuances.

Netflix makes a far more obvious sin: not having “who is watching” as boolean choices. If I am watching with my partner, I want both of our accounts to mark that series as viewed. And I really want Netflix to tell me what I’m watching with her so that I don’t continue watching it without her because I will be single if that happens (again).

It would be a great revenue stream for Netflix.

Are you sure you want to watch this without your partner ?

Yes ? We recommend the following service for finding temporary accommodation on short notice

Re: ML is not that good at predicting consumers' choices

#64

It is the same on Netflix. I have phases where I watch a certain genre for a few weeks and then move on. For example after a few Scandi crime series it is time for something else. However, at the same time my daughter loves Animé and pretty only watch that. It is really hard for an ML algorithm to grab these nuances.

No post body was provided.

Re: ML is not that good at predicting consumers' choices

#65

Always interesting to see outsiders writing papers about this, using anecdote and unrelated data (mostly political and real world purchase data in this case) to argue that ML doesn't make useful predictions. Meanwhile I look at randomized controlled trial data showing millions of dollars in revenue uplift directly attributable to ML vs non-ML backed conversion pipelines, offsetting the cost of doing the ML by >10x. I…

Exactly, RCTs take the mystery out. Nice work!

Re: ML is not that good at predicting consumers' choices

#66
> for most of the more interesting consumer decisions, those that are “new” and non-habitual, prediction remains hard

Translation: Computers can't read minds.

A bigger generalization is that, whenever a software feature becomes essentially mind reading; someone's either feeding a hype engine or letting their imagination run away.

The best things to do in that case is to pop the bubble if you can, or walk away. I will often clearly state, "Computers can't read minds. You're making a lot of assumptions that will most likely prove false."

Re: ML is not that good at predicting consumers' choices

#67

Always interesting to see outsiders writing papers about this, using anecdote and unrelated data (mostly political and real world purchase data in this case) to argue that ML doesn't make useful predictions. Meanwhile I look at randomized controlled trial data showing millions of dollars in revenue uplift directly attributable to ML vs non-ML backed conversion pipelines, offsetting the cost of doing the ML by >10x. I…

Are you really sure you're not just fooling yourselves with your randomized controlled trials? As Feynman famously said, the easiest person to fool is yourself. And in business even more than science, you might even like the results. Have you ever put this data up against something similar to the peer review system in academia, where several experts from a competing deparment (or ideally competing company) try to pic…

well, certainly it's possible to fool yourselves with A/B testing, it doesn't mean you must be fooling yourselves. I've also seen similar results in recommendation settings in mobile gaming, not once but over and over again across portfolio of dozens of games/hundreds millions of players. You don't need to predict 20% better on whatever you are predicting to get a 20% increase in LTV and it's even better if you are doing RL since you are optimizing directly for your KPIs

Re: ML is not that good at predicting consumers' choices

#68

Always interesting to see outsiders writing papers about this, using anecdote and unrelated data (mostly political and real world purchase data in this case) to argue that ML doesn't make useful predictions. Meanwhile I look at randomized controlled trial data showing millions of dollars in revenue uplift directly attributable to ML vs non-ML backed conversion pipelines, offsetting the cost of doing the ML by >10x. I…

> Always interesting to see outsiders writing papers about this, using anecdote and unrelated data (mostly political and real world purchase data in this case) to argue that ML doesn't make useful predictions. Meanwhile I look at randomized controlled trial data showing millions of dollars in revenue uplift directly attributable to ML vs non-ML backed conversion pipelines, offsetting the cost of doing the ML by >10x.

I regularly buy the same brand of toilet paper, socks, and sneakers. Machine learning can predict that.

But, machine learning can't predict that I spent the night at my parents house, really liked the fancy pillow they put on the guest bed, and then had to buy one for myself. (This is essentially the conclusion in the abstract.)

Such a prediction requires mind reading, which is impossible.

Re: ML is not that good at predicting consumers' choices

#69
post #53

Always interesting to see outsiders writing papers about this, using anecdote and unrelated data (mostly political and real world purchase data in this case) to argue that ML doesn't make useful predictions. Meanwhile I look at randomized controlled trial data showing millions of dollars in revenue uplift directly attributable to ML vs non-ML backed conversion pipelines, offsetting the cost of doing the ML by >10x. I…

If your ML model is able to predict what consumers are going to buy, the revenue lift would be zero. Let's say I go to the store to buy milk. The store has a perfect ML model, so they're able to predict that I'm about to do that. I walk into the store and buy the milk as planned. So how does the ML help drive revenue? The store could make my life easier by having it ready for me at the door, but I was going to buy it…

It wouldn't be zero. If you wanted milk but couldn't find it in the store/spent too much, you might just give up on buying it.

Re: ML is not that good at predicting consumers' choices

#70
post #68

Always interesting to see outsiders writing papers about this, using anecdote and unrelated data (mostly political and real world purchase data in this case) to argue that ML doesn't make useful predictions. Meanwhile I look at randomized controlled trial data showing millions of dollars in revenue uplift directly attributable to ML vs non-ML backed conversion pipelines, offsetting the cost of doing the ML by >10x. I…

> Always interesting to see outsiders writing papers about this, using anecdote and unrelated data (mostly political and real world purchase data in this case) to argue that ML doesn't make useful predictions. Meanwhile I look at randomized controlled trial data showing millions of dollars in revenue uplift directly attributable to ML vs non-ML backed conversion pipelines, offsetting the cost of doing the ML by >10x.…

The key insight missed by this paper (and people from the marketing field in general) is that cases like that are extremely rare compared to easy to predict cases. They don't matter right now at all for most products, from the perspective of marketing ROI.

Also ML can predict that, BTW. Facebook knows you are connected to your parents. If the pillow seller tells Facebook that your parents bought the pillow, then Facebook knows and may choose to show you an ad for that pillow.

Post reply on HN