Evaluating Search Algorithms
shopify.engineering
Evaluating Search Algorithms
1–10 of 18 posts
Re: Evaluating Search Algorithms
#2Great article!
This seems like a fairly tricky ranking function. I wonder if they compared it to combining TF-IDF and the page popularity. This would help with the problem they explained.
It'd be interesting to see more details about how they implemented the query-specific page rank.
Re: Evaluating Search Algorithms
#3I should re-read the article because I can't see what kind of problem they try to solve with MAP, NDCG and "invented here" Pagerank what couldn't be solved with tf-idf and out-of-the box Elasticsearch functionality. It's a highly underrated peace of software.
Re: Evaluating Search Algorithms
#4Interesting article! Shopify's approach is cool, it's interesting they're using Kafka to generate datasets. I wonder if the explicit human rankings will get stale (and also be hugely outweighted by implicit judgements in the training data). The real-time feedback aspect sounds cool, I wonder if it's just for metrics or also for re-training in real-time.
I worked on a Learning To Rank implementation a year or so ago. What struck me then (and now reading about Shopify's implementation) is that the approach is often very similar across sites, but the implementation is usually rather tailored. You see the same patterns: online/offline metrics; nDCG; click models and implicit/explicit relevance judgements; re-ranking top-k of results, and so on.
Unfortunately there doesn't seem to be a technology tying all of the components of an LtR system together. A managed service like Algolia could be an answer. I wonder if industry will eventually converge on a framework, such as an extension to Open Source Connection's Elasticsearch Learning to Rank plugin (https://diff.wikimedia.org/2017/10/17/elasticsearch-learning...).
It's a really interesting area of theory and practice - I hope Shopify write more about their implementation!
I'd also recommend reading Airbnb's really excellent paper - https://arxiv.org/pdf/1810.09591.pdf.
Re: Evaluating Search Algorithms
#5I should re-read the article because I can't see what kind of problem they try to solve with MAP, NDCG and "invented here" Pagerank what couldn't be solved with tf-idf and out-of-the box Elasticsearch functionality. It's a highly underrated peace of software.
Hi, I worked on improving the search rankings for a popular package manager. Imagine your search algorithm is already excellent, you have countless of documents, and customers do countless of unique queries. Now say that you want to improve your search rankings further. How do you do that? What if your improvement helps some queries but hurts others? Things like tf-idf or Elasticsearch won't help here.
That's where NDCG comes in! Basically it gives a score for your search rankings that you can use to compare different search algorithms. The higher the score, the closer your algorithm was to producing the expected search results. This is super useful as you can try lots of experiments and get a good sense of whether the experiment is promising or not.
Re: Evaluating Search Algorithms
#6Great article! This seems like a fairly tricky ranking function. I wonder if they compared it to combining TF-IDF and the page popularity. This would help with the problem they explained. It'd be interesting to see more details about how they implemented the query-specific page rank.
Great question! We intended the article to be more of an introduction to the evaluation process for search algos and definitely glossed over the example's problem area here and should have talked more about the specific experiment we used as the example. We use TF-IDF in combination with the popularity scores for both algorithms mentioned. There are a few reasons we chose to experiment with the query-specific popularity - mainly, that global popularity wasn't as much of a signal for "helpfulness" for the user's intent as much as it was a signal for the most popular pages overall (i.e. turns out to be overview pages and most popular problems encountered on the platform), so it wasn't having the desired effect on ranking. We wanted to see whether using popularity within the context of the user's topic enhanced the utility of the popularity score. Plus, with Query-Specific ranking, when the user's keywords aren't an exact match with the document's language, we still receive a boost from users who previously scrolled further to find the match to solve their problem
Re: Evaluating Search Algorithms
#7Interesting article! Shopify's approach is cool, it's interesting they're using Kafka to generate datasets. I wonder if the explicit human rankings will get stale (and also be hugely outweighted by implicit judgements in the training data). The real-time feedback aspect sounds cool, I wonder if it's just for metrics or also for re-training in real-time. I worked on a Learning To Rank implementation a year or so ago.…
Appreciate the feedback and recommendation! You're right that explicit judgments can get stale - fortunately for our document collection the information architecture and article structures themselves are slow-changing (the answers themselves might change, but the document that answers the question probably won't for some time). We also primarily use explicit judgments to label head queries/common topics, and may augment our datasets with fresh data from time to time. The team is currently exploring augmenting these human-made datasets with automatic judgments using click models.
For realtime feedback, we've implemented (on another search product at Shopify, not the Help Center) a "near"-time feedback loop using implicit judgments to alter search results. Perhaps I'll write a post about that one soon :) . My colleague Doug talks a bit about the new systems we're building in this blog post - https://shopify.engineering/apache-beam-for-search-getting-s....
Re: Evaluating Search Algorithms
#8Interesting article! Shopify's approach is cool, it's interesting they're using Kafka to generate datasets. I wonder if the explicit human rankings will get stale (and also be hugely outweighted by implicit judgements in the training data). The real-time feedback aspect sounds cool, I wonder if it's just for metrics or also for re-training in real-time. I worked on a Learning To Rank implementation a year or so ago.…
Appreciate the feedback and recommendation! You're right that explicit judgments can get stale - fortunately for our document collection the information architecture and article structures themselves are slow-changing (the answers themselves might change, but the document that answers the question probably won't for some time). We also primarily use explicit judgments to label head queries/common topics, and may augm…
Great stuff, and it’s cool you are working with Doug, I enjoyed his book on search relevance :) I’ll look out for more posts from your team, good luck!
Re: Evaluating Search Algorithms
#9Where do the relevance scores come from? Are they human rated? I feel like that could leave room for error as raters would probably not have the same opinion as me on what a good document is.
Re: Evaluating Search Algorithms
#10[deleted]