Live data from Hacker News

Building a Simple Search Engine That Works

karboosx.net

81–82 of 82 posts

Re: Building a Simple Search Engine That Works

#81
post #79
post #63

Earlier quoted context omitted.

There are more kinds of search engines than just internet search engines. At this point I’m is almost certain that the non-internet search engines of the world are much larger than internet search engines. Edit: And I’m getting downvoted for this. If it’s because I am tangential to the original comment then that’s fair. If it’s because you think I’m wrong, I have worked on the two largest internet search engines in t…

What do you mean by a non-internet search engine, and what might be one that is bigger than Google/Bing?

You’ve got to remember that google/bing do not index the internet entire. Part of their magic is selectively indexing only a tiny sliver and still being effective.

Other kinds of search systems have to index everything, which simplifies things but has its own scaling challenges.

Easiest way to think about it is that while the majority of webpages are never indexed, every blob of text in a social media post, private message in an app, email, document, etc in every major app in the world, including the ones with billions of users, is indexed in a search engine for that app:

- GSuite search (think of how many gmails are searchable in the world right now… and they are all indexed)

- the enterprise search powering ChatGPT, Claude (these maybe there by now, if not they are likely well on the way)

- The Microsoft 365 search (this is probably massive with so many corporate email systems and teams systems on it)

- slack search

- X(twitter) search

- ticktock search (this idk, I’ve never used ticktock but if every video and every comment is searchable then this is probably huge)

- Facebook search (especially since this is likely combined across its product suite)

These are probably all larger in effective size than google or bing.

Re: Building a Simple Search Engine That Works

#82

Earlier quoted context omitted.

Alright but why do we not have more search engines that are actually good? I'd love to cut myself off from Google, including Google Search, but any alternatives manage to be even worse. Consistently so. It's as if Google won the war by being just permanently slightly better - while everyone is actually really crap. That wasn't the case, say, 10 years ago or so.

Not all search is web-wide search. The best-known example of this is probably Amazon's search bar. No one really wants to search Amazon via Google. They have staffers contributing heavily to Lucene. But also there are all kinds of other applications. Let's say you run a reviews site; you can build a bespoke power search form allowing people to sort on things like price, date of review, set a minimum star threshhold,…

Before Lucene, search was really hard
Post reply on HN