love the style, colors and the cookie popup from https://karboosx.net/.
Anyone knows if its an open source framework/style/tool being used here or it just the web dev skills of the author that are superb?
Building a Simple Search Engine That Works
41–50 of 82 posts
Re: Building a Simple Search Engine That Works
#42The idea behind search itself is very simple, and it's a fun problem domain that I encourage anyone to explore[1]. The difficulties in search are almost entirely dealing with the large amounts of data, both logistically and in handling underspecified queries. A DBMS-backed approach breaks down surprisingly fast. Probably perfectly fine if you're indexing your own website, but will likely choke on something the size o…
What is the order of magnitude of the largest document store that you can practically work from SQLite on a single thousand-dollar server run by some text-heavy business process? For text search, roughly how big of a corpus can we practically search if we're occupying... let's say five seconds per query, twelve queries per minute?
Re: Building a Simple Search Engine That Works
#43The idea behind search itself is very simple, and it's a fun problem domain that I encourage anyone to explore[1]. The difficulties in search are almost entirely dealing with the large amounts of data, both logistically and in handling underspecified queries. A DBMS-backed approach breaks down surprisingly fast. Probably perfectly fine if you're indexing your own website, but will likely choke on something the size o…
> The difficulties in search are almost entirely dealing with the large amounts of data, both logistically and in handling underspecified queries. I would expect the difficulty to be deciding which item to return when there are multiple that contain the search term. Is wikipedia's article on Gilligan's Island better than some guy's blog post? Or is that guy a fanatic who has spent his entire life pondering whether Wr…
Elastic and many others fail to solve this problem too. There are many different strategies and many of them require ingenuity and development.
Re: Building a Simple Search Engine That Works
#44Earlier quoted context omitted.
Do you have a source how funding yandex funds the war? Yandex is a great search engine, so I would hate to find out that this is true
It's based in Russia so it presumably pays taxes and salaries in Russia.
All American companies pay taxes to America which is basically always commiting atrocities so I don't think that's a strogn enough reason on its own.
Re: Building a Simple Search Engine That Works
#45Building a simple text search engine isn't that hard. People show them off on HN on a fairly regular basis. Most of those are fairly primitive. Unfortunately building a good search engine isn't that straightforward. There's more to it than just implementing bm25 (the goto ranking algorithm), which you can vibe code in a few minutes these days. The reason this is easy is because this is nineties era research that is a…
what do you think about ManticoreSearch? It has been around longer than Lucene
Re: Building a Simple Search Engine That Works
#46Earlier quoted context omitted.
> The difficulties in search are almost entirely dealing with the large amounts of data, both logistically and in handling underspecified queries. I would expect the difficulty to be deciding which item to return when there are multiple that contain the search term. Is wikipedia's article on Gilligan's Island better than some guy's blog post? Or is that guy a fanatic who has spent his entire life pondering whether Wr…
Elastic and many others fail to solve this problem too. There are many different strategies and many of them require ingenuity and development.
It’s not like ElasticSearch lacks ranking algorithms and control thereof. But it can require tuning and adjustment for various domains. Relevancy is, after all, subjective.
Re: Building a Simple Search Engine That Works
#47why isn't there a place to post something where someone else will find it when searching that doesn't require auth? i get the logistics of what i'm asking, but i really think we need a global index.
Re: Building a Simple Search Engine That Works
#48The idea behind search itself is very simple, and it's a fun problem domain that I encourage anyone to explore[1]. The difficulties in search are almost entirely dealing with the large amounts of data, both logistically and in handling underspecified queries. A DBMS-backed approach breaks down surprisingly fast. Probably perfectly fine if you're indexing your own website, but will likely choke on something the size o…
> The difficulties in search are almost entirely dealing with the large amounts of data, both logistically and in handling underspecified queries.
Large amounts of data seem obviously difficult.
For your second difficulty, "handling underspecified queries": it seems to me that's a subset of the problem of, "given a query, what are the most relevant results?" That problem seems very tricky, partially because there is no exact true answer.
marginalia search is great as a contrast to engines like google, in part because google chooses to display advertisements as the most relevant results.
Have you found any of the TREC papers helpful?
Re: Building a Simple Search Engine That Works
#49Re: Building a Simple Search Engine That Works
#50Building a simple text search engine isn't that hard. People show them off on HN on a fairly regular basis. Most of those are fairly primitive. Unfortunately building a good search engine isn't that straightforward. There's more to it than just implementing bm25 (the goto ranking algorithm), which you can vibe code in a few minutes these days. The reason this is easy is because this is nineties era research that is a…
> "I don't need all that fancy stuff". Yes you do.
> let's turn off dynamic mapping and not index all those text fields you never query on