Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Successful machine learning models: lessons learned at Booking.com

blog.acolyer.org

Post reply on HN
Print view View original

81–90 of 93 posts

Re: Successful machine learning models: lessons learned at Booking.com

#81

Post by otabdeveloper4 »

nerdponx wrote: post #28
minxomat wrote: post #18

Earlier quoted context omitted.

That gave me a flashback to the early 2000s AIML chatbot craze. "What, you have an online store and no customer chatbot? What are you doing ?!"

Flipside is that this ML (whether you consider it AI or not) really is delivering huge value to the businesses that deploy them.

Not really.

Today's ML is really good at speech and image recogniziton, which makes for some very eye-popping layman demos.

Whereas what businesses really want is time series prediction, and modern ML really sucks balls at solving this problem.

Re: Successful machine learning models: lessons learned at Booking.com

#82

Post by jiofih »

romaaeterna wrote: post #80
orasis wrote: post #79

Earlier quoted context omitted.

If you’re making decisions that involve multiple variables you may be doing hundreds to thousands of inferences for a single page load. Keeping latency under 50ms becomes a real challenge.

But it comes down to this doesn't it: x1 * a + x2 * b + x3 * c + ... + x1000 * zzz + ... If a, b, c ... zzz, are all fixed constants already discovered by your learning algorithm. That's a very fast calculation, and doesn't take anything like 50ms. Also, in the real world, you can establish a significance cutoff for a lot of these constants and get something like this as your final equation: x13 * m + x523 * cdf + x7…

The inputs to those functions might be coming from external data sources, or aggregated. These have a cost too. But mostly, it just adds up. At a thousand features, you have a 0.05ms budget for each. Without taking into account network latency since you won’t be running those models inside the application server.

Re: Successful machine learning models: lessons learned at Booking.com

#83

Post by romaaeterna »

jiofih wrote: post #82
romaaeterna wrote: post #80

Earlier quoted context omitted.

But it comes down to this doesn't it: x1 * a + x2 * b + x3 * c + ... + x1000 * zzz + ... If a, b, c ... zzz, are all fixed constants already discovered by your learning algorithm. That's a very fast calculation, and doesn't take anything like 50ms. Also, in the real world, you can establish a significance cutoff for a lot of these constants and get something like this as your final equation: x13 * m + x523 * cdf + x7…

The inputs to those functions might be coming from external data sources, or aggregated. These have a cost too. But mostly, it just adds up. At a thousand features, you have a 0.05ms budget for each. Without taking into account network latency since you won’t be running those models inside the application server.

So why not load the calculated constants to the application server to reduce network latency?

And the learning side of things should have culled that list of thousand features down to a list of 5 - 10 that mattered.

It really sounds like the off-the-shelf stuff isn't built for efficiency.

Re: Successful machine learning models: lessons learned at Booking.com

#84

Post by rhizome »

jhncls wrote: post #32
Bootwizard wrote: post #20

Earlier quoted context omitted.

What do you mean by "dark patterns"? I'm not familiar with that term

Ben Edelman (Harvard, Microsoft) published a study [0] about how dark patterns in the online travel industry help them reach margins up to 25%. He also mentions the consolidation where most well-known booking sites are owned by just two large groups. [0] http://www.benedelman.org/impact-of-ota-bias-and-consolidati...

Sounds like travel agents are becoming more and more cost effective every day. I mean, what does the Hyatt website do that a phone call 30 years ago couldn't?

Re: Successful machine learning models: lessons learned at Booking.com

#85

Post by [deleted] »

romaaeterna wrote: post #83
jiofih wrote: post #82

Earlier quoted context omitted.

The inputs to those functions might be coming from external data sources, or aggregated. These have a cost too. But mostly, it just adds up. At a thousand features, you have a 0.05ms budget for each. Without taking into account network latency since you won’t be running those models inside the application server.

So why not load the calculated constants to the application server to reduce network latency? And the learning side of things should have culled that list of thousand features down to a list of 5 - 10 that mattered. It really sounds like the off-the-shelf stuff isn't built for efficiency.

[deleted]

Re: Successful machine learning models: lessons learned at Booking.com

#86

Post by chudi »

joker3 wrote: post #22
benmaraschino wrote: post #19

Earlier quoted context omitted.

Not the OP, but I work on similar problems, albeit in a different setting (healthcare, millions+ of patients). The gist is that you have to bake experimental design into the deployment of your ML model, but in many cases a simple RCT or A/B test just won't cut it. This is largely because when you deploy a model, no matter how sophisticated or accurate, there's no guarantee that it'll actually move the needle in terms…

Do you know a good introduction to adaptive/sequential designs? I'm looking for something along the lines of a textbook aimed at a graduate level seminar.

take a look at this https://www.amazon.com/Multi-armed-Bandit-Allocation-Indices...

Re: Successful machine learning models: lessons learned at Booking.com

#87

Post by chudi »

sandGorgon wrote: post #58

>> Booking.com go to some lengths to minimise the latency introduced by models, including horizontally scaled distributed copies of models, a in-house developed custom linear prediction engine, favouring models with fewer parameters, batching requests, and pre-computation and/or caching. Any idea what these are ? especially the pre-computation/caching and batching. I'm not able to see what advantage does batching bri…

In my company we use this approach, instead of infer online, we just run all our models overnight and save the results on a database that we serve trough an API, that gives you constant time on the latency, its a shotgun approach as much of the recommendations aren't served (specially if are user facing and not item facing ones) but works really well.

All our models are balanced using multi-armed bandits, so for our recommendations engine, we run lots of arms that depends on the incoming channel, were in the app is being shown the recommendation, etc and just combine the outputs of the models.

Re: Successful machine learning models: lessons learned at Booking.com

#88

Post by nerdponx »

otabdeveloper4 wrote: post #81
nerdponx wrote: post #28

Earlier quoted context omitted.

Flipside is that this ML (whether you consider it AI or not) really is delivering huge value to the businesses that deploy them.

Not really. Today's ML is really good at speech and image recogniziton, which makes for some very eye-popping layman demos. Whereas what businesses really want is time series prediction, and modern ML really sucks balls at solving this problem.

Deep learning or machine learning?

I agree on the former, and quite strongly disagree on the latter, even if it means redefining ML to be dressed-up statistics.

Re: Successful machine learning models: lessons learned at Booking.com

#89

Post by sangnoir »

softwaredoug wrote: post #76
sangnoir wrote: post #57

Earlier quoted context omitted.

> what do they think will happen? Booking.com A/B tests everything : the drill-sergeant-like funnel probably has higher conversion rates than any gentler variation. So to answer your question - they think you might not book through them without the shoutiness.

But does it have higher customer retention over the long term? Conversions are only one piece of running an online business

> But does it have higher customer retention over the long term?

I'm going to guess the answer is no - which is why organizations have to be careful which metrics they measure and incentivize on. Granted, this failing is industry-wide as the longest view on most orgs' dashboards is YoY. When that metric starts freefalling, it most likely will be too late to do anything about it, but most of the staff (up to the CEO) will have padded Resumes with amazing numbers for improved conversion/revenue which will get them to the next job.

Re: Successful machine learning models: lessons learned at Booking.com

#90

Post by otabdeveloper4 »

nerdponx wrote: post #88
otabdeveloper4 wrote: post #81

Earlier quoted context omitted.

Not really. Today's ML is really good at speech and image recogniziton, which makes for some very eye-popping layman demos. Whereas what businesses really want is time series prediction, and modern ML really sucks balls at solving this problem.

Deep learning or machine learning? I agree on the former, and quite strongly disagree on the latter, even if it means redefining ML to be dressed-up statistics.

Forecasting in 2019 is still using techniques from 1950. You can call that "machine learning", but only if you really want to make people confused by marketing speak.
Post reply on HN