Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a Simple Search Engine That Works

karboosx.net

Post reply on HN
Print view View original

51–60 of 82 posts

Re: Building a Simple Search Engine That Works

#51

Post by wheybags »

19l21 wrote: post #18
shevy-java wrote: post #10

Earlier quoted context omitted.

Alright but why do we not have more search engines that are actually good? I'd love to cut myself off from Google, including Google Search, but any alternatives manage to be even worse. Consistently so. It's as if Google won the war by being just permanently slightly better - while everyone is actually really crap. That wasn't the case, say, 10 years ago or so.

I use the '4get' proxy search engine, which lets you use pretty much every search engine under the sun, for both websites and images. It's really useful because it is faster than google, and if you need to find some pages you can just change the search engine quickly. It is open source and there are many instances available, I use '4get.bloat.cat' or '4get.lunar.iu' It is a better alternative to SearX for sure

I checked the about page on 4get.bloat.cat, and within the first paragraph of the "what is this" section, it used the phrase "globohomo bullshit". I dont think these are people I want to support.

Re: Building a Simple Search Engine That Works

#52

Post by MrAlex94 »

SyneRyder wrote: post #30
MrAlex94 wrote: post #20

Earlier quoted context omitted.

Not quite independent as it’s a meta-search, but I developed a subscription based one at search.waterfox.net. Pays for the infrastructure costs and remains ad/tracking free.

Nice! I couldn't see the list of search engines that are included in your meta-search, the FAQ currently seems to imply that it only serves Google results? If you give users the option to include / not include certain search engines in their results, so their money never goes to those particular engine companies, that could be of interest to some Kagi refugees. I ended up vibe coding my own meta-search engine (augmen…

Currently only Google, but Brave and Mojeek are going to be made available as well very soon

Re: Building a Simple Search Engine That Works

#53

Post by DanielHB »

puilp0502 wrote: post #22
DanielHB wrote: post #15

Earlier quoted context omitted.

People who work on really obscure things love to talk about their work, heck if someone would listen to me I could talk for hours about what I do. Unfortunately very few people care about the minutia of making a behemoth system work.

I would. Heck, I bet half of HN would be interested in what kind of insanity lies under those behemoths.

I work with music streaming, it is mostly just a lot of really banal business rules that become an entangled web of convoluted if statements. Where to show a single button might mean hitting 5 different microservices and checking 10 different booleans

Re: Building a Simple Search Engine That Works

#54

Post by marginalia_nu »

mapt wrote: post #42
marginalia_nu wrote: post #21

The idea behind search itself is very simple, and it's a fun problem domain that I encourage anyone to explore[1]. The difficulties in search are almost entirely dealing with the large amounts of data, both logistically and in handling underspecified queries. A DBMS-backed approach breaks down surprisingly fast. Probably perfectly fine if you're indexing your own website, but will likely choke on something the size o…

What is the order of magnitude of the largest document store that you can practically work from SQLite on a single thousand-dollar server run by some text-heavy business process? For text search, roughly how big of a corpus can we practically search if we're occupying... let's say five seconds per query, twelve queries per minute?

If you held a gun to my head and forced me to make a guess I'd say you could push that approach to order of 100K, maybe 1M documents.

If sqlite had a generic "strictly ascending sequence of integers" type[1] and would optimize around that, you could probably push it farther in terms of implementing efficient inverted indexes.

[1] primary key tables aren't really useful here.

Re: Building a Simple Search Engine That Works

#55

Post by wink »

My pet peeve for search engines for content I use is that they regularly ignore 2-letter and 3-letter "words" or acronyms. If all I need is a search for "mp3" then stripping exactly that is not useful ;) (was just the first file extension that came to my mind, but "PHP" works just as well).

Re: Building a Simple Search Engine That Works

#56

Post by zipy124 »

marginalia_nu wrote: post #21

The idea behind search itself is very simple, and it's a fun problem domain that I encourage anyone to explore[1]. The difficulties in search are almost entirely dealing with the large amounts of data, both logistically and in handling underspecified queries. A DBMS-backed approach breaks down surprisingly fast. Probably perfectly fine if you're indexing your own website, but will likely choke on something the size o…

I think in today's world the harder problem is evading SEO spam. A search engine is in constant war with adverserarial players, who need you to see their content for revenue, rather than the actual answer.

This neccessitates a constant game of cat and mouse, where you adjust your quality metric so SEO shops can't figure it out and capitalise on it.

Re: Building a Simple Search Engine That Works

#58

Post by wolfgarbe »

jillesvangurp wrote: post #29
internet_points wrote: post #28

Earlier quoted context omitted.

Seems like good advice, search has been built quite a few times now :-) I've defaulted to elasticsearch myself. However, have you tried running any of the "up and coming" alternatives that keep showing up here? In particular, https://github.com/SeekStorm/SeekStorm seems very interesting, though I haven't heard from anyone using it in prod.

A red flag for me is that it lists stopword lists as a feature. Those went out of fashion in Lucene/Elasticsearch because of some non trivial but very effective caching and other optimizations around version 5. Stopwords are an old school optimization to deal with the problem of high frequency tokens when calculating rankings. Basically that means dealing with long lists of document ids that e.g. contain the word "to…

The stopword list in SeekStorm is purely optional, per default it is empty.

The query "to be or not to be" that you mentioned, consisting solely of stopwords, returns complete results and perform quite well in the benchmark: https://github.com/SeekStorm/SeekStorm?tab=readme-ov-file#be...

Both Lucene and Elastic still offer stopword filters: https://lucene.apache.org/core/10_3_2/analysis/common/org/ap... https://www.elastic.co/docs/reference/text-analysis/analysis...

Re: Building a Simple Search Engine That Works

#59

Post by zppln »

zipy124 wrote: post #56
marginalia_nu wrote: post #21

The idea behind search itself is very simple, and it's a fun problem domain that I encourage anyone to explore[1]. The difficulties in search are almost entirely dealing with the large amounts of data, both logistically and in handling underspecified queries. A DBMS-backed approach breaks down surprisingly fast. Probably perfectly fine if you're indexing your own website, but will likely choke on something the size o…

I think in today's world the harder problem is evading SEO spam. A search engine is in constant war with adverserarial players, who need you to see their content for revenue, rather than the actual answer. This neccessitates a constant game of cat and mouse, where you adjust your quality metric so SEO shops can't figure it out and capitalise on it.

I feel at this point you'd almost be better off hand-curating a set of domains and only crawl those.

Re: Building a Simple Search Engine That Works

#60

Post by jayd16 »

zipy124 wrote: post #56
marginalia_nu wrote: post #21

The idea behind search itself is very simple, and it's a fun problem domain that I encourage anyone to explore[1]. The difficulties in search are almost entirely dealing with the large amounts of data, both logistically and in handling underspecified queries. A DBMS-backed approach breaks down surprisingly fast. Probably perfectly fine if you're indexing your own website, but will likely choke on something the size o…

I think in today's world the harder problem is evading SEO spam. A search engine is in constant war with adverserarial players, who need you to see their content for revenue, rather than the actual answer. This neccessitates a constant game of cat and mouse, where you adjust your quality metric so SEO shops can't figure it out and capitalise on it.

I wonder how hard it is when mice are not paying the cat to serve ads.
Post reply on HN