71–80 of 82 posts
Earlier quoted context omitted.
I feel at this point you'd almost be better off hand-curating a set of domains and only crawl those.
not sure if this was intentional, but everything old is new again; back to OH yahoo? or Craig's list?
I think SEO plam + AI slop is likely to lead us back to human curation.
About a decade ago, I was working with a guy who was getting a PhD in search engine design, which I knew/know nothing about. It was actually a lot of fun to chat with him, because he was so enthusiastic about how searching works and how it can integrate with databases, and he was eager to explain this all to anyone who would listen. I learned a fair amount from him, though admittedly I still don't know much about the…
People who work on really obscure things love to talk about their work, heck if someone would listen to me I could talk for hours about what I do. Unfortunately very few people care about the minutia of making a behemoth system work.
It’s so easy to be cynical and not care about anything, I am certainly guilty of that. Older people who have found things that they can truly geek out about for hours are relatively rare and some of my favorite people as a result (and part of the reason that I like going to conferences).
I like my coworkers and they’re certainly not anti-intellectual or anything, but there’s only so long I can ramble on about TLA+ or Isabelle or Alloy before they lose interest. It’s not a fault on them at all, there are plenty of topics I am not interested in.
Long time ago, I've really enjoyed a course by David Evans from Virginia University about building a search engine and concepts of computer science. Building a "classic" search engine is a very fun project to go through. https://www.cs.virginia.edu/~evans/courses/cs101/ List: https://www.youtube.com/watch?v=9nkR2LLPiYo&list=PLAwxTw4SYa... Dave's profile: https://www.cs.virginia.edu/~evans/
Earlier quoted context omitted.
This was the solution I was thinking about, but I thought, well that's the way someone would have done it 20 years ago
Alright but why do we not have more search engines that are actually good? I'd love to cut myself off from Google, including Google Search, but any alternatives manage to be even worse. Consistently so. It's as if Google won the war by being just permanently slightly better - while everyone is actually really crap. That wasn't the case, say, 10 years ago or so.
Earlier quoted context omitted.
What is the order of magnitude of the largest document store that you can practically work from SQLite on a single thousand-dollar server run by some text-heavy business process? For text search, roughly how big of a corpus can we practically search if we're occupying... let's say five seconds per query, twelve queries per minute?
If you held a gun to my head and forced me to make a guess I'd say you could push that approach to order of 100K, maybe 1M documents. If sqlite had a generic "strictly ascending sequence of integers" type[1] and would optimize around that, you could probably push it farther in terms of implementing efficient inverted indexes. [1] primary key tables aren't really useful here.
But it is true that a specialized search engine using a more clever algorithm might be another order of magnitude faster.
Earlier quoted context omitted.
Do you have a source how funding yandex funds the war? Yandex is a great search engine, so I would hate to find out that this is true
It's based in Russia so it presumably pays taxes and salaries in Russia.
Earlier quoted context omitted.
I think in today's world the harder problem is evading SEO spam. A search engine is in constant war with adverserarial players, who need you to see their content for revenue, rather than the actual answer. This neccessitates a constant game of cat and mouse, where you adjust your quality metric so SEO shops can't figure it out and capitalise on it.
I wonder how hard it is when mice are not paying the cat to serve ads.
Earlier quoted context omitted.
What is the order of magnitude of the largest document store that you can practically work from SQLite on a single thousand-dollar server run by some text-heavy business process? For text search, roughly how big of a corpus can we practically search if we're occupying... let's say five seconds per query, twelve queries per minute?
If you held a gun to my head and forced me to make a guess I'd say you could push that approach to order of 100K, maybe 1M documents. If sqlite had a generic "strictly ascending sequence of integers" type[1] and would optimize around that, you could probably push it farther in terms of implementing efficient inverted indexes. [1] primary key tables aren't really useful here.
Is that not what WITHOUT ROWID does? My understanding is that it's precisely meant to physically cluster data in the underlying B-Tree
If that is not what you meant, could you elaborate on the "primary key tables aren't really useful here" footnote?
Earlier quoted context omitted.
I think in today's world the harder problem is evading SEO spam. A search engine is in constant war with adverserarial players, who need you to see their content for revenue, rather than the actual answer. This neccessitates a constant game of cat and mouse, where you adjust your quality metric so SEO shops can't figure it out and capitalise on it.
There are more kinds of search engines than just internet search engines. At this point I’m is almost certain that the non-internet search engines of the world are much larger than internet search engines. Edit: And I’m getting downvoted for this. If it’s because I am tangential to the original comment then that’s fair. If it’s because you think I’m wrong, I have worked on the two largest internet search engines in t…
Building a simple text search engine isn't that hard. People show them off on HN on a fairly regular basis. Most of those are fairly primitive. Unfortunately building a good search engine isn't that straightforward. There's more to it than just implementing bm25 (the goto ranking algorithm), which you can vibe code in a few minutes these days. The reason this is easy is because this is nineties era research that is a…