Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ML is not that good at predicting consumers' choices

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

71–80 of 228 posts

Re: ML is not that good at predicting consumers' choices

#71

Post by capableweb »

bryanrasmussen wrote: post #62
Slackwise wrote: post #10

Earlier quoted context omitted.

"Why am I sweating right now? Oh, the Nest set the temperature too high again!" And then after a few instances, I just turn off all the automation and set up a schedule like normal. Same with the "away from home" which seems to randomly think I'm away and I have no idea why. Oh, and the app doesn't show me filter reminders, only the actual device, which I never touch all the way downstairs. There's not even any statu…

>And then after a few instances, I just turn off all the automation and set up a schedule like normal. If you have a fairly regular life I would think a schedule would outdo ML pretty much all the time, because you know exactly what that schedule should be. ML might be useful for a secret agent whose life is so erratic that a schedule would be useless. That is to say ML is maybe better than falling back to nothing.

If your schedule is so irregular/erratic, how is a ML algorithm supposed to be able to learn it?

Sounds like in that case it's better to just control things manually.

Re: ML is not that good at predicting consumers' choices

#72

Post by ape4 »

im3w1l wrote: post #36
ape4 wrote: post #22

You just bought a washing machine... could I interest you in a washing machine?

GPT can solve this! I prompted it with "Sarah bought a washing machine and a ". It completed "dryer.". Another "If you buy a hammer you might also want to buy " -> "a nail". Ill forgive the singular. Just to be clear those are not cherry picked - they were my first two attempts.

Putting those together... I actually bought a pair of anti hammer arrestors for the washing machine ;)

Re: ML is not that good at predicting consumers' choices

#73

Post by bschne »

abirch wrote: post #50
mgraczyk wrote: post #42

Always interesting to see outsiders writing papers about this, using anecdote and unrelated data (mostly political and real world purchase data in this case) to argue that ML doesn't make useful predictions. Meanwhile I look at randomized controlled trial data showing millions of dollars in revenue uplift directly attributable to ML vs non-ML backed conversion pipelines, offsetting the cost of doing the ML by >10x. I…

Amazon does a remarkably good job of predicting what I'll buy and I frequently add to my purchases.

I know this is slightly off what the article is concerned with, but the important question in a business context is whether this prediction is worth anything, i.e. whether it can be turned into revenue that wouldn't be generated in the absence of the prediction.

Re: ML is not that good at predicting consumers' choices

#74

Post by bryanrasmussen »

capableweb wrote: post #71
bryanrasmussen wrote: post #62

Earlier quoted context omitted.

>And then after a few instances, I just turn off all the automation and set up a schedule like normal. If you have a fairly regular life I would think a schedule would outdo ML pretty much all the time, because you know exactly what that schedule should be. ML might be useful for a secret agent whose life is so erratic that a schedule would be useless. That is to say ML is maybe better than falling back to nothing.

If your schedule is so irregular/erratic, how is a ML algorithm supposed to be able to learn it? Sounds like in that case it's better to just control things manually.

ML can learn patterns that humans might not be aware of, so you there might be certain things that happen that show you will be on a mission to East Asia for a couple days.

Re: ML is not that good at predicting consumers' choices

#75

Post by malkia »

Some years ago, I worked on a team "Ads Human Eval" - we had raters hired to do A/B testing for ads. These evaluated questionaires carefuly crafted by our linguists, and then analyzed by the statisticians providing feedback to the (internal) group that wanted to know more about.

So the best experience was this internal event that we had, where the raters would say that certain Ad would not fare well (long term), while the initial metrics (automated) were showing the opposite (short temr). So then we'll gather into this event, and people would "debug" these and try to find where the differences are coming through.

Then we had to help another group, where ML failed miserably detecting ads that should've not been shown on specific media, and raters came to help giving the correct answers.

The one thing that I've learned is that humans are not going to be replaced any time soon by AI, and I've been telling my folks, friends or anyone (new-born luddities) - that automation is not going to fully replace us. We'll still be needed as teachers, evaluators, fixers, tweakers/hackers - e.g. someone saying - this is right, and this is not, this needs adjustment, etc. (to the machine, ai, etc.).

Maybe machines are going to take over us one day, but until then, I'm not worried...

(I've also understood I knew nothing about staticics, and how valuable linguists are when comes to forming clear, concise and non-confusing (no double meaning) questions)

Re: ML is not that good at predicting consumers' choices

#76

Post by jeffreyrogers »

Plough_Jogger wrote: post #26

This review omits techniques from reinforcement learning (especially bandits) that have been used successfully in industry for years now.

How are bandits used in consumer choice problems? Bandits solve almost the inverse problem: which choice to offer/take when it's uncertain which is best, but the problem under consideration in the blog post is about predicting which choice a consumer will pick, a standard marketing problem.

Re: ML is not that good at predicting consumers' choices

#77

Post by conformist »

mgraczyk wrote: post #42

Always interesting to see outsiders writing papers about this, using anecdote and unrelated data (mostly political and real world purchase data in this case) to argue that ML doesn't make useful predictions. Meanwhile I look at randomized controlled trial data showing millions of dollars in revenue uplift directly attributable to ML vs non-ML backed conversion pipelines, offsetting the cost of doing the ML by >10x. I…

Yes, the review paper appears to be roughly conditioned on "using data that academics can readily access or generate".

Clearly, this doesn't generalise to cases where you have highly specific data (e.g. if you're Google).

However, cases with large societal impact are more likely to be the latter? They may perhaps better be viewed as "conditioned on data that is so valuable that nobody is going to publish or explain it", which kind of is in the complement of the review?

Re: ML is not that good at predicting consumers' choices

#79

Post by mrxd »

soared wrote: post #57
mrxd wrote: post #53

Earlier quoted context omitted.

If your ML model is able to predict what consumers are going to buy, the revenue lift would be zero. Let's say I go to the store to buy milk. The store has a perfect ML model, so they're able to predict that I'm about to do that. I walk into the store and buy the milk as planned. So how does the ML help drive revenue? The store could make my life easier by having it ready for me at the door, but I was going to buy it…

This is incorrect. You can predict many things that drive incremental revenue lift. The simplest: Predict what features a user is most interested in, drive them to that page (increasing their predicted conversion rate) -> purchases that occur now that would not have occurred before. Similarly: Predict products a user is likely to purchase given they made a different purchase. The user may not have seen these incremen…

ML is useful for many things. I'm asking the question of whether prediction is useful, and whether it is accurate to describe ML as making predictions.

The reason to raise those questions is that for many people, the word prediction has connotations of surveillance and control, so it is best not to use it loosely.

The meaning of the word "predict" is to indicate a future event, so it doesn't make grammatical sense to put a present tense verb after it, as you have done in "Predict what features a user is most interested in." Aside from the verb being in the present tense, being interested in something is not an event.

You can't predict a present state of affairs. If I look out the window and see that it is raining, no one would say that I've predicted the weather. If I come to that conclusion indirectly (e.g. a wet umbrella by the door), that would not be considered a prediction either because it's in the present. The accurate term for this is "inference", not "prediction".

The usage of the word predict is also incorrect from the point of view of an A/B test. If your ML model has truly predicted that your users will purchase a particular product, they will purchase it regardless of which condition they are in. But this is the null hypothesis, and the ML model is being introduced in the treatment group to disprove this.

Re: ML is not that good at predicting consumers' choices

#80

Post by macNchz »

I've shared this before on HN, but it never fails to make me laugh when I think about it:

>Several years ago a conversation about a similar topic prompted me to look at the ad targeting data Facebook had on me. At the time I'd had a Facebook account for 12 years with lots of posts, group memberships and ~500 friends. Their cutting edge data collection and complex ad targeting algorithms had identified my "Hobbies and activities" as: "Mosquito", "Hobby", "Leaf" and "Species": https://imgur.com/nWCWn63. Whatever that means.

Post reply on HN