Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What software engineers should know about search (2017)

scribe.rip

Post reply on HN
Print view View original

101–110 of 132 posts

Re: What software engineers should know about search (2017)

#101

Post by dotancohen »

danpalmer wrote: post #96
dotancohen wrote: post #76

Earlier quoted context omitted.

One strategy for this is to parse the search string for facets. Presumably there would be a facet for "t-shirt" and another facet for "blue", modulo synonyms and typos. Selecting those facets should give very good results, even without hitting the full-text search.

Yes that's a good option, but it still requires that you've got the facets in the index. I'm surprised at how many people say "just use postgress full-text search" without thinking through what information will actually be made available to search, for example.

No, I'm not saying to have the facets in the index. I'm saying to parse the search text for facets, and to search on the facets.

If the whole string matches facets, then the full text search never gets touched, as in the example above. But if the example were e.g. "blue anime t-shirt" then you could facet on "blue" and "t-shirt" while FTS only has to contend with "anime".

Re: What software engineers should know about search (2017)

#102

Post by outlawBand »

dotancohen wrote: post #69
kristopolous wrote: post #2

Biggest advice I can give is you probably don't need search if you're indexable by search bots. No really. Look over people's shoulders sometime. They'll just go to Google and type in their search followed by terms such as Wikipedia, imdb, Stackoverflow, YouTube, Bandcamp, Amazon, eBay, Yelp... all sites that spent a lot of time on their search and have done quite a decent job. Oh well. So unless you really need it f…

> Biggest advice I can give is you probably don't need search if you're indexable by search bots. I don't know how relevant it would be today, but ~2005 or so I had a site that used Google search for the internal "search" functionality. It was a webmaster feature of Google at the time, the google results would be displayed with my site's branding, colours, etc. Only results from my site were displayed, but there was…

So as a casual web browser, I have been guilty of this. IMHO it just breaks the users experience when they feel like they have been swept out to google. Almost like when someone asks a question, and someone replies with a link from lmgtfy

Re: What software engineers should know about search (2017)

#103

Post by svcrunch »

BiteCode_dev wrote: post #4
kristopolous wrote: post #2

Biggest advice I can give is you probably don't need search if you're indexable by search bots. No really. Look over people's shoulders sometime. They'll just go to Google and type in their search followed by terms such as Wikipedia, imdb, Stackoverflow, YouTube, Bandcamp, Amazon, eBay, Yelp... all sites that spent a lot of time on their search and have done quite a decent job. Oh well. So unless you really need it f…

Yes, but a good embeded search makes all the diff. VueJS and Tailwind CSS both are indexed, but on those particular site I use the web site search _when I'm looking back for the reference of something I know_, because it's faster and more accurate than googling. Granted, it's rare, but if you manage it, it's great.

Agree with your sentiment.

I cofounded ZIR AI to provide ML (vector)-based search as a PaaS solution, similar to what Algolia or Elasticsearch do for keyword searching.

We have a demo (https://zir-ai.com/demo) running over Quanta Magazine articles. Not only can it outperform the keyword search embedded on quantamagazine.org, but one some queries, it even outperforms Google with a site restrict (e.g. how old is the universe site:quantamagazine.org).

> Granted, it's rare, but if you manage it, it's great.

ML-powered search will make this commonplace, once the tech goes mainstream in the next 2-3 years.

Re: What software engineers should know about search (2017)

#104

Post by hihihihi1234 »

hihihihi1234 wrote: post #63
sriram_malhar wrote: post #57

Earlier quoted context omitted.

Would it bother you if foo.com took your query and send "query site:foo.com" to google? The results would be foo.com specific.

I often find this gives better results. Eg I'm much likely to find what I'm looking for if I search for "(query) site:reddit.com" than if I use reddit's own search feature.

Much more likely*

Re: What software engineers should know about search (2017)

#105

Post by abecedarius »

sydthrowaway wrote: post #43
kristopolous wrote: post #2

Biggest advice I can give is you probably don't need search if you're indexable by search bots. No really. Look over people's shoulders sometime. They'll just go to Google and type in their search followed by terms such as Wikipedia, imdb, Stackoverflow, YouTube, Bandcamp, Amazon, eBay, Yelp... all sites that spent a lot of time on their search and have done quite a decent job. Oh well. So unless you really need it f…

An interesting example is the search query “ reddit”. Without the ‘reddit’ qualifier, the results are nearly always spammy and useless (as much as modern Reddit tries to compete here notwithstanding)

Reddit itself has an annoying dark-patterns UI. You have to edit the URL into old.reddit.com to get a reasonable page to read.

Re: What software engineers should know about search (2017)

#106

Post by dotancohen »

outlawBand wrote: post #102
dotancohen wrote: post #69

Earlier quoted context omitted.

> Biggest advice I can give is you probably don't need search if you're indexable by search bots. I don't know how relevant it would be today, but ~2005 or so I had a site that used Google search for the internal "search" functionality. It was a webmaster feature of Google at the time, the google results would be displayed with my site's branding, colours, etc. Only results from my site were displayed, but there was…

So as a casual web browser, I have been guilty of this. IMHO it just breaks the users experience when they feel like they have been swept out to google. Almost like when someone asks a question, and someone replies with a link from lmgtfy

I wish that we had spoken a decade and a half ago ))

Re: What software engineers should know about search (2017)

#107

Post by cratermoon »

netcan wrote: post #46
kristopolous wrote: post #2

Biggest advice I can give is you probably don't need search if you're indexable by search bots. No really. Look over people's shoulders sometime. They'll just go to Google and type in their search followed by terms such as Wikipedia, imdb, Stackoverflow, YouTube, Bandcamp, Amazon, eBay, Yelp... all sites that spent a lot of time on their search and have done quite a decent job. Oh well. So unless you really need it f…

This is true in principle, but in practice, the use of 3rd party search has died down over the last decade. This is not the phenomenon of a superior way winning out. Either search is not an important feature, and a suboptimal, DIY implementation that looks OK is good enough. Or, search is a primary feature and then you need control over it. IE, if you have an online store, travel site or dating app with a search base…

> controlling UIs matters the most

Or as former Google design ethicist Tristan Harris wrote, "Whoever controls the menu controls the choices."

https://observer.com/2016/06/how-technology-hijacks-peoples-...

Re: What software engineers should know about search (2017)

#108

Post by woolion »

kristopolous wrote: post #91
woolion wrote: post #51

Earlier quoted context omitted.

Is there any actual evidence that might be true? I encountered this argument many times, but only from programmers (who thought it was to difficult because of their tech stack), never from users. As a user, there is no site that does not have search, no matter how good it is, that does not feel quite horrible because of that. Even the search by Google feel horrible -- romhacking comes to mind. It is also frustrating…

The statement is intentionally hedging and qualitative. Some people do the usage pattern, some don't. The advocacy is to push back on search as a difficult and often unnecessary problem to solve. The caveat is my increasingly toxic pattern to push back on almost everything as unnecessary. It's probably overly antagonistic. I'm a techno pessimist programmer. I didn't understand this attitude when I was younger but the…

Formulated like that, I'm very sympathetic to your point of view. Having a small set of well-refined features rather than a growing spaghetti of half done ones. I guess my own point of contention is that programmers would say "task X is hard" when it isn't that much, it is that their technical choices make it hard.

For instance, one Front-End engineer was really proud of using a very recent and trendy framework to rewrite the whole site from scratch. After deployment, people are ordering the wrong products. His reply was, "keeping the query filters in sync is too hard of a problem". It's really not.

Re: What software engineers should know about search (2017)

#109

Post by jamesbriggs »

Being from 2017 the article misses some of the coolest advances in semantic search, which is now pretty easy and lets you search in (almost) the same way they would ask a shop assistant when looking for something specific - "do you know where the thing with the cool circles and pointy bits is?" (maybe being a little more specific...)

Google do this and they're very good at it, but a lot of companies need their own search capabilities - think about those internal help pages. They usually seem super outdated compared to the semantic search capabilities of Google.

In the end there's only a few components to it, you use some NLP model to create what are called 'dense vectors'. Then you put all these dense vectors into an 'index' which is optimized for fast search (that comes under the umbrella of ANN search). Then given a new query you just compare that to the items in the index and return the most similar results.

I covered the search part of it (https://www.pinecone.io/learn/) with Pinecone, who provide managed-search - although here we mainly focus on Faiss (a great engine from Facebook AI). However, we're also looking to create some content covering the first half too, which is 'how to build dense vectors' using models like BERT, we have one post so far on that (https://www.pinecone.io/learn/dense-vector-embeddings-nlp/)

Re: What software engineers should know about search (2017)

#110

Post by jamesbriggs »

hnbad wrote: post #68
amir wrote: post #29

Not sure who needs to know this but scribe.rip is a relatively new alternative reader for Medium. The original article is at: https://medium.com/startup-grind/what-every-software-enginee... HN's special treatment of medium.com links doesn't apply to scribe ones.

That explains why all the link texts are weirdly misaligned. Do the authors have any input or does it just scrape Medium for content?

A lot of websites scrape Medium without the authors having any idea, I assume it is another one of those
Post reply on HN