> Model performance is not the same as business performance This is interesting. Sometimes some people from business side consider that AI is the solution to all problems (as if there was just one catch-them-all AI solution) and some academic people think that the top-performance model for some classification task is the must-go, and all they forget that the goal is to earn money.
People should be using more than one design, but that's not as commonly taught as it should be. I'm going to give a talk about that to my company's ML group in about a month, and hopefully that'll improve things where I am.
Not sure if you'd be willing, but I'd love a quick rundown of the high level takeaways if you'd be willing to drop them here. Are you talking about more than one experimental design in terms of comparing the exp/control distributions or something else?
Experimental design is just a t-test? At least accord to that picture it seems that way. There are no ANOVA or interaction test? Do websites usually just use t-test only? Like adding one feature at a time?
It's even worst than that. Most of the time the validity of the t-test that they are running is questionable. They are technically running an online t-test and as soon as they find significance they stop. This is fundamentally wrong, and not conclusive at all.
That was an interesting result from the original Netflix challenge. First of all it turned out that the winner wasn't actually all that useful for various reasons such as computational intensity. But, more interestingly, it also turned out that the goals of the model--"best" recommendations--isn't actually the goal of Netflix at all which is much more interested in customer retention and similar metrics. The two thin…
The big problem for Netflix was that their data was all from DVD rentals, but by the end of the contest, their business model was very streaming oriented. As you might imagine, people have beliefs about what they'll want to watch in a few days that don't exactly match up with what they want to watch now. That difference killed the model.
another problem was that their "user" was actually a household, with a range of (sometimes conflicting) likes and interests - which could result in strange recommendations. They finally added profiles to fix that data problem.
Not sure if you'd be willing, but I'd love a quick rundown of the high level takeaways if you'd be willing to drop them here. Are you talking about more than one experimental design in terms of comparing the exp/control distributions or something else?
I'm just going to present the absolute basics of the topic. It'll be a high-level overview of something along the lines of chapters 1-4 of https://www.amazon.com/Analysis-Experiments-Chapman-Statisti... .
As an aside, > developing an organisational capability to design, build, and deploy successful machine learned models in user-facing contexts is, in my opinion, as fundamental to an organisation’s competitiveness You hear that, right? In 2019 already you have to have AI and do it well to be competitive. I just wanted to point out how cyberpunk that is.
That gave me a flashback to the early 2000s AIML chatbot craze. "What, you have an online store and no customer chatbot? What are you doing?!"
People should be using more than one design, but that's not as commonly taught as it should be. I'm going to give a talk about that to my company's ML group in about a month, and hopefully that'll improve things where I am.
Not sure if you'd be willing, but I'd love a quick rundown of the high level takeaways if you'd be willing to drop them here. Are you talking about more than one experimental design in terms of comparing the exp/control distributions or something else?
Not the OP, but I work on similar problems, albeit in a different setting (healthcare, millions+ of patients). The gist is that you have to bake experimental design into the deployment of your ML model, but in many cases a simple RCT or A/B test just won't cut it. This is largely because when you deploy a model, no matter how sophisticated or accurate, there's no guarantee that it'll actually move the needle in terms of the outcomes you care about—hence you need to run some kind of trial. At the same time, you want to maximize overall utility by not having to allocate more subjects to your control arm (or harmful, or resource-intensive and ineffective treatment arms) than you need to. This latter point is much more of a problem in medicine than it is in other settings, as you can imagine. These considerations point to adaptive designs that balance exploration/exploitation, e.g. those based on multi-armed bandits. Currently working on some cool (in my opinion) variations of MABs that incorporate domain-specific knowledge, so I could talk about this all day!
"Content Overload: Accommodations have very rich content, e.g.descriptions, pictures, reviews and ratings." Laughed at that one. Booking.com is so full of dark patterns that I dread using it.
What do you mean by "dark patterns"? I'm not familiar with that term