Successful machine learning models: lessons learned at Booking.com • Hacker News Board

Re: Successful machine learning models: lessons learned at Booking.com

#91

Post by tgsovlerkhgsel » Wed, Oct 09, 2019, 11:59 PM UTC

ohduran wrote: post #67

duxup wrote: post #52

Earlier quoted context omitted.

I'll get the phrasing wrong here but "What is easy to measure will be deemed important what is difficult to measure will be deemed unimportant."

Oh, absolutely! How can you measure "I'm sick of being tricked into buying", "I wish this site treated me with respect", "I'm the client, and I feel like the product"?

They do/did have a customer satisfaction survey somewhere at the end or after the booking process.

I made sure to provide feedback.

When people book with you but give you 1/10 stars, that's probably a pretty strong warning sign that the customer isn't happy with the site and the first usable competitor that they find will get their business.

Re: Successful machine learning models: lessons learned at Booking.com

#92

Post by Breza » Wed, Oct 16, 2019, 5:28 PM UTC

romaaeterna wrote: post #83

jiofih wrote: post #82

Earlier quoted context omitted.

The inputs to those functions might be coming from external data sources, or aggregated. These have a cost too. But mostly, it just adds up. At a thousand features, you have a 0.05ms budget for each. Without taking into account network latency since you won’t be running those models inside the application server.

So why not load the calculated constants to the application server to reduce network latency? And the learning side of things should have culled that list of thousand features down to a list of 5 - 10 that mattered. It really sounds like the off-the-shelf stuff isn't built for efficiency.

Nobody's saying this is an impossible problem. The paper shows how much additional work is required beyond a traditional data science workflow.

The team behind the paper built a model that had good performance on training data. They're a smart lot so they knew they needed to cross-validate. The results held up in cross-validation! Hooray, the model works! ...right?

That's as far as a lot of data scientists go. This paper points out that you need to have a model that does (at least) three things: 1. Generates good scores with training and testing data 2. Outperforms existing models in the real world 3. Runs really really quickly There are a lot of data scientists who have no idea how to do #2 and #3. This paper says "These parts are really important!!!"

Re: Successful machine learning models: lessons learned at Booking.com

#93

Post by Breza » Wed, Oct 16, 2019, 5:30 PM UTC

chudi wrote: post #86

joker3 wrote: post #22

Earlier quoted context omitted.

Do you know a good introduction to adaptive/sequential designs? I'm looking for something along the lines of a textbook aimed at a graduate level seminar.

take a look at this https://www.amazon.com/Multi-armed-Bandit-Allocation-Indices...

Thanks!