Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a Simple Search Engine That Works

karboosx.net

Post reply on HN
Print view View original

61–70 of 82 posts

Re: Building a Simple Search Engine That Works

#61

Post by jillesvangurp »

wolfgarbe wrote: post #58
jillesvangurp wrote: post #29

Earlier quoted context omitted.

A red flag for me is that it lists stopword lists as a feature. Those went out of fashion in Lucene/Elasticsearch because of some non trivial but very effective caching and other optimizations around version 5. Stopwords are an old school optimization to deal with the problem of high frequency tokens when calculating rankings. Basically that means dealing with long lists of document ids that e.g. contain the word "to…

The stopword list in SeekStorm is purely optional, per default it is empty. The query "to be or not to be" that you mentioned, consisting solely of stopwords, returns complete results and perform quite well in the benchmark: https://github.com/SeekStorm/SeekStorm?tab=readme-ov-file#be... Both Lucene and Elastic still offer stopword filters: https://lucene.apache.org/core/10_3_2/analysis/common/org/ap... https://www.e…

Thanks for correcting me and clarifying this.

Re: Building a Simple Search Engine That Works

#62

Post by jillesvangurp »

nchmy wrote: post #45
jillesvangurp wrote: post #25

Building a simple text search engine isn't that hard. People show them off on HN on a fairly regular basis. Most of those are fairly primitive. Unfortunately building a good search engine isn't that straightforward. There's more to it than just implementing bm25 (the goto ranking algorithm), which you can vibe code in a few minutes these days. The reason this is easy is because this is nineties era research that is a…

what do you think about ManticoreSearch? It has been around longer than Lucene

I have no experience with ManticoreSearch but they've been around for a while. I think it migh be a Sphinx spinoff, this was a long abandoned solr like search engine written in C++ that they seem to have forked (correct me if I'm wrong). Mainly popular for some ecommerce use cases (as is the case with Solr). Looking at their front page I don't see any compelling reason to switch and a couple of things that I don't like

- GPLv3 better than AGPLv3 in Elasticsearch but less permissive than Apache 2.0 in Opensearch.

- They seem to emphasize being a drop in replacement a lot. Which raises the question: why not just stick with Opensearch.

- I'm very skeptical of benchmarks in this space. Mostly they are apples and oranges comparisons. As I argued earlier it mainly raises the question what they are not doing or skipping. Barring major algorithmic improvements which Lucene developers could just copy if it's valid, I don't see how they could be better/faster. And Lucene is of course well known to be heavily optimized and still squeezing out a lot of performance from release to release. Progress has been pretty substantial in v8 and v9 in recent years.

Other than that they seem to know what they are doing is the best I can say about it.

Re: Building a Simple Search Engine That Works

#63

Post by HEmanZ »

zipy124 wrote: post #56
marginalia_nu wrote: post #21

The idea behind search itself is very simple, and it's a fun problem domain that I encourage anyone to explore[1]. The difficulties in search are almost entirely dealing with the large amounts of data, both logistically and in handling underspecified queries. A DBMS-backed approach breaks down surprisingly fast. Probably perfectly fine if you're indexing your own website, but will likely choke on something the size o…

I think in today's world the harder problem is evading SEO spam. A search engine is in constant war with adverserarial players, who need you to see their content for revenue, rather than the actual answer. This neccessitates a constant game of cat and mouse, where you adjust your quality metric so SEO shops can't figure it out and capitalise on it.

There are more kinds of search engines than just internet search engines. At this point I’m is almost certain that the non-internet search engines of the world are much larger than internet search engines.

Edit: And I’m getting downvoted for this. If it’s because I am tangential to the original comment then that’s fair. If it’s because you think I’m wrong, I have worked on the two largest internet search engines in the world and one non-internet search engine that dwarfed both in size (although different in complexity).

Re: Building a Simple Search Engine That Works

#64

Post by nchmy »

jillesvangurp wrote: post #62
nchmy wrote: post #45

Earlier quoted context omitted.

what do you think about ManticoreSearch? It has been around longer than Lucene

I have no experience with ManticoreSearch but they've been around for a while. I think it migh be a Sphinx spinoff, this was a long abandoned solr like search engine written in C++ that they seem to have forked (correct me if I'm wrong). Mainly popular for some ecommerce use cases (as is the case with Solr). Looking at their front page I don't see any compelling reason to switch and a couple of things that I don't li…

Yeah, it forked from sphinx when sphinx died or rugpulled or something like that. Seems to be very actively maintained.

I think one of the main draws is that it is a single binary rather than the complexity of ES/OS, JVM etc... And if you have a mysql/mariadb database, it just connects and automatically ingests extremely quickly. They also use Galera for replication, but I also think its not as explicitly shareded, which simplifies things.

Yeah, their benchmarks are astounding, so much so that it is hard to believe. yet, I have seen them be quite open to feedback, collaboration etc so

Anyway, thanks for your thoughts and insights!

Re: Building a Simple Search Engine That Works

#65

Post by eduction »

shevy-java wrote: post #10
aorloff wrote: post #4

Earlier quoted context omitted.

This was the solution I was thinking about, but I thought, well that's the way someone would have done it 20 years ago

Alright but why do we not have more search engines that are actually good? I'd love to cut myself off from Google, including Google Search, but any alternatives manage to be even worse. Consistently so. It's as if Google won the war by being just permanently slightly better - while everyone is actually really crap. That wasn't the case, say, 10 years ago or so.

Not all search is web-wide search. The best-known example of this is probably Amazon's search bar. No one really wants to search Amazon via Google. They have staffers contributing heavily to Lucene.

But also there are all kinds of other applications. Let's say you run a reviews site; you can build a bespoke power search form allowing people to sort on things like price, date of review, set a minimum star threshhold, etc. You can also weigh product names or review titles more heavily in the index scoring (a review /of/ the Pixel 10 should rank higher than a review that mentions the Pixel 10 prominently).

Even being able to sort results of searching blog posts or other dated content by date is powerful - Google can only guess at the actual dates of those posts. You can search with required tags, or weigh tags more heavily in result scoring. You can put your finger on the scale and say, effectively, post A should always rank more highly than post B for term X.

Also, site operators know traffic/popularity, which internet search engine can only sort of guess at, and can use this to score/sort. Amazon clearly does this.

For some reason a lot of web devs seem to think search is this really hard problem. But once you learn the basics of how it works, and if you use a library like Lucene, it does not need to be hard at all. Mostly you just have to be strategic and consistent about where and when you index and deindex content, it's usually alongside your db persistence calls. Once it's running you optimize by sprinkling some minimum amount of magic on your scoring setup to make it worthwhile/differentiated from Google.

Re: Building a Simple Search Engine That Works

#66

Post by franczesko »

Long time ago, I've really enjoyed a course by David Evans from Virginia University about building a search engine and concepts of computer science.

Building a "classic" search engine is a very fun project to go through.

https://www.cs.virginia.edu/~evans/courses/cs101/

List: https://www.youtube.com/watch?v=9nkR2LLPiYo&list=PLAwxTw4SYa...

Dave's profile: https://www.cs.virginia.edu/~evans/

Re: Building a Simple Search Engine That Works

#67

Post by vjvjvjvjghv »

DanielHB wrote: post #15
tombert wrote: post #5

About a decade ago, I was working with a guy who was getting a PhD in search engine design, which I knew/know nothing about. It was actually a lot of fun to chat with him, because he was so enthusiastic about how searching works and how it can integrate with databases, and he was eager to explain this all to anyone who would listen. I learned a fair amount from him, though admittedly I still don't know much about the…

People who work on really obscure things love to talk about their work, heck if someone would listen to me I could talk for hours about what I do. Unfortunately very few people care about the minutia of making a behemoth system work.

It seems a common problem in our profession that you can’t really talk to anybody about what you are doing. My friends have a vague idea but that’s it.

Re: Building a Simple Search Engine That Works

#68

Post by vjvjvjvjghv »

dyml wrote: post #12
ku1ik wrote: post #11

Earlier quoted context omitted.

Try kagi.com. I tried and stayed. It’s paid though.

I also used Kagi, but decided to cancel my subscription last year when it was revealed they pay Yandex for their search, which is a Russian company that ultimately fuels the Russian war on Ukraine. Once Kagi stops transferring money to Russia, I’d be happy re-subscribe.

I have the feeling that, if you look a little closer, a lot of products you are using are supporting atrocities somewhere directly or indirectly.

Re: Building a Simple Search Engine That Works

#69

Post by skeeter2020 »

zppln wrote: post #59
zipy124 wrote: post #56

Earlier quoted context omitted.

I think in today's world the harder problem is evading SEO spam. A search engine is in constant war with adverserarial players, who need you to see their content for revenue, rather than the actual answer. This neccessitates a constant game of cat and mouse, where you adjust your quality metric so SEO shops can't figure it out and capitalise on it.

I feel at this point you'd almost be better off hand-curating a set of domains and only crawl those.

not sure if this was intentional, but everything old is new again; back to OH yahoo? or Craig's list?

Re: Building a Simple Search Engine That Works

#70

Post by [deleted] »

HEmanZ wrote: post #63
zipy124 wrote: post #56

Earlier quoted context omitted.

I think in today's world the harder problem is evading SEO spam. A search engine is in constant war with adverserarial players, who need you to see their content for revenue, rather than the actual answer. This neccessitates a constant game of cat and mouse, where you adjust your quality metric so SEO shops can't figure it out and capitalise on it.

There are more kinds of search engines than just internet search engines. At this point I’m is almost certain that the non-internet search engines of the world are much larger than internet search engines. Edit: And I’m getting downvoted for this. If it’s because I am tangential to the original comment then that’s fair. If it’s because you think I’m wrong, I have worked on the two largest internet search engines in t…

[deleted]
Post reply on HN