Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What every software engineer should know about search

medium.com

Post reply on HN
Print view View original

11–20 of 57 posts

Re: What every software engineer should know about search

#11

Post by jventura »

I used to work on textual Information Retrieval (which is quite related to search). One of the key contributions of my PhD thesis was a statistical method to extract concepts from texts [0].

Surprisingly enough, with those concepts, Tf-Idf [1] was quite good to extract keywords from documents which allowed us to build document descriptor tables which can eventually be used for document search [2]. We also built a small prototype that allowed us to retrieve results of specific areas of documents where the searched concepts were more "valuable" than on other parts of the document [3].

It is a very interesting subarea of AI/NLP which unfortunately doesn't seem to attract much interest.

Since the article also talks a bit about Wikipedia dumps datasets, here's a tool that I've created to build textual corpora: https://github.com/joaoventura/WikiCorpusExtractor

[0] - http://www.sciencedirect.com/science/article/pii/S1877050912...

[1] - https://en.wikipedia.org/wiki/Tf%E2%80%93idf

[2] - https://link.springer.com/chapter/10.1007/978-3-642-40669-0_...

[3] - https://link.springer.com/chapter/10.1007/978-3-642-40669-0_...

Re: What every software engineer should know about search

#12

Post by amelius »

mss2 wrote: post #8

I think somewhere along the way work on search just devolved into handling millions of queries at a time. It could have been so much better. If you ask a "search expert" today what's he is trying to fix, he will say something related to scaling. If you asked an expert in the 80's or 90's they would talk about query complexity and NLP i.e. Who were the four semi-finalists of last years Wimbledon? And you would get bac…

Well, I hear a lot of people complaining that the results on DuckDuckGo are still worse than on Google, even though both search-engines produce results within a second. And these are people that really want to quit using Google for privacy reasons. I never hear people complaining that a search is slow. So I do think that search-quality is where the competition is happening.

Edit: But, I agree, we don't often see any good HN posts or papers about search-quality. Perhaps a case of trade secrets?

Re: What every software engineer should know about search

#13

Post by mayank »

makmanalp wrote: post #4
nostrademons wrote: post #2

Ex-Google search engineer here, now using hosted ElasticSearch extensively in my startup. This is a really good overview. If there's one part I want to highlight, it's that you should expect to spend a lot of time fine-tuning your ranking function for your particular product & corpus . The default ElasticSearch ranking function kinda sucks. It was changed in ES 5.0 to Okapi BM25, which is the current academic state-o…

> When you use a service with good search that just works, chances are that there's been a lot of engineer hours devoted to identifying exactly which signals are most useful in your corpus I imagine at a place like Google, your "corpus" is just about everything under the sun, as are your queries. (i.e. less chance for a subject-specific tuning) What happens then?

> I imagine at a place like Google, your "corpus" is just about everything under the sun, as are your queries. > What happens then?

You accept the fact that a good search engine is inevitably going to be a multi-layered beast that requires constant effort to improve, with heuristics, machine learning, magic constants, special lists of words, and regular expressions.

After that, it's just a matter of picking a commercially useful objective function ("instant search", "query autocomplete", "maximize user satisfaction") and pointing large teams of well-resourced PhDs at it. Probably a whole weekend at least.

Re: What every software engineer should know about search

#14

Post by cpburns2009 »

metafunctor wrote: post #10
cpburns2009 wrote: post #6

What's with the random unicode characters spread throughout this article? Several headings and sentences are prefixed oddly with an exclamation mark ("\u2757\ufe0f"), or diamond ("\U0001f537").

There's an “Emoji Legend“ near the top of the article. It's a nice idea, flagging part of the text with coarse semantic meaning. However, I find I don't trust the writer's judgement to decide what is important for me; I must read the whole thing anyway.

I've never encountered an "Emoji Legend" before. I instinctively skip over the word "emoji" so I missed that section.

Re: What every software engineer should know about search

#17

Post by binarymax »

mss2 wrote: post #8

I think somewhere along the way work on search just devolved into handling millions of queries at a time. It could have been so much better. If you ask a "search expert" today what's he is trying to fix, he will say something related to scaling. If you asked an expert in the 80's or 90's they would talk about query complexity and NLP i.e. Who were the four semi-finalists of last years Wimbledon? And you would get bac…

Actually quite the opposite. While we do make sure performance is good, the bulk of our time is spent dealing with relevance improvements, content wrangling, language pipelines, and lots of other cool stuff.

Re: What every software engineer should know about search

#18

Post by jimnotgym »

genpfault wrote: post #15
jimnotgym wrote: post #7

Very interesting article. However I really hate the floating menu and footer on the site. I browse on a small laptop and I really don't like the obscured viewing

Seemed OK with Reader View in Firefox.

It is OK with Reader View , it's not really the point though is it?

Re: What every software engineer should know about search

#20

Post by garysieling »

makmanalp wrote: post #4
nostrademons wrote: post #2

Ex-Google search engineer here, now using hosted ElasticSearch extensively in my startup. This is a really good overview. If there's one part I want to highlight, it's that you should expect to spend a lot of time fine-tuning your ranking function for your particular product & corpus . The default ElasticSearch ranking function kinda sucks. It was changed in ES 5.0 to Okapi BM25, which is the current academic state-o…

> When you use a service with good search that just works, chances are that there's been a lot of engineer hours devoted to identifying exactly which signals are most useful in your corpus I imagine at a place like Google, your "corpus" is just about everything under the sun, as are your queries. (i.e. less chance for a subject-specific tuning) What happens then?

If you can identify categories of queries or documents that interest people, you can treat tuning that part of the search engine as if it was a new product.
Post reply on HN