Successful machine learning models: lessons learned at Booking.com
1–10 of 93 posts
Re: Successful machine learning models: lessons learned at Booking.com
#2> Model performance is not the same as business performance
This is interesting. Sometimes some people from business side consider that AI is the solution to all problems (as if there was just one catch-them-all AI solution) and some academic people think that the top-performance model for some classification task is the must-go, and all they forget that the goal is to earn money.
Re: Successful machine learning models: lessons learned at Booking.com
#3> Model performance is not the same as business performance This is interesting. Sometimes some people from business side consider that AI is the solution to all problems (as if there was just one catch-them-all AI solution) and some academic people think that the top-performance model for some classification task is the must-go, and all they forget that the goal is to earn money.
That was an interesting result from the original Netflix challenge.
First of all it turned out that the winner wasn't actually all that useful for various reasons such as computational intensity.
But, more interestingly, it also turned out that the goals of the model--"best" recommendations--isn't actually the goal of Netflix at all which is much more interested in customer retention and similar metrics. The two things may be correlated but they're certainly not the same thing.
I don't remember all the details but I thought it was a really good insight at the time.
Re: Successful machine learning models: lessons learned at Booking.com
#4[deleted]
Re: Successful machine learning models: lessons learned at Booking.com
#5As an aside,
> developing an organisational capability to design, build, and deploy successful machine learned models in user-facing contexts is, in my opinion, as fundamental to an organisation’s competitiveness
You hear that, right? In 2019 already you have to have AI and do it well to be competitive. I just wanted to point out how cyberpunk that is.
Re: Successful machine learning models: lessons learned at Booking.com
#6> Model performance is not the same as business performance This is interesting. Sometimes some people from business side consider that AI is the solution to all problems (as if there was just one catch-them-all AI solution) and some academic people think that the top-performance model for some classification task is the must-go, and all they forget that the goal is to earn money.
> the goal is to earn money
Yep, from my experience with booking.com it seems that instead of using highly trained AIs the decision was made to simply slap every dark pattern known to man onto the site and auto-subscribe every customer to a dozen newsletters.
Re: Successful machine learning models: lessons learned at Booking.com
#7Experimental design is just a t-test? At least accord to that picture it seems that way. There are no ANOVA or interaction test?
Do websites usually just use t-test only? Like adding one feature at a time?
Re: Successful machine learning models: lessons learned at Booking.com
#8Experimental design is just a t-test? At least accord to that picture it seems that way. There are no ANOVA or interaction test? Do websites usually just use t-test only? Like adding one feature at a time?
People should be using more than one design, but that's not as commonly taught as it should be. I'm going to give a talk about that to my company's ML group in about a month, and hopefully that'll improve things where I am.
Re: Successful machine learning models: lessons learned at Booking.com
#9> Model performance is not the same as business performance This is interesting. Sometimes some people from business side consider that AI is the solution to all problems (as if there was just one catch-them-all AI solution) and some academic people think that the top-performance model for some classification task is the must-go, and all they forget that the goal is to earn money.
That was an interesting result from the original Netflix challenge. First of all it turned out that the winner wasn't actually all that useful for various reasons such as computational intensity. But, more interestingly, it also turned out that the goals of the model--"best" recommendations--isn't actually the goal of Netflix at all which is much more interested in customer retention and similar metrics. The two thin…
The big problem for Netflix was that their data was all from DVD rentals, but by the end of the contest, their business model was very streaming oriented. As you might imagine, people have beliefs about what they'll want to watch in a few days that don't exactly match up with what they want to watch now. That difference killed the model.
Re: Successful machine learning models: lessons learned at Booking.com
#10Experimental design is just a t-test? At least accord to that picture it seems that way. There are no ANOVA or interaction test? Do websites usually just use t-test only? Like adding one feature at a time?
People should be using more than one design, but that's not as commonly taught as it should be. I'm going to give a talk about that to my company's ML group in about a month, and hopefully that'll improve things where I am.
Not sure if you'd be willing, but I'd love a quick rundown of the high level takeaways if you'd be willing to drop them here.
Are you talking about more than one experimental design in terms of comparing the exp/control distributions or something else?