51–60 of 735 posts
> If you search for "Plato", you might for example end up at the Canterbury Tales. Go looking for the Canterbury Tales, and you may stumble upon Neil Gaiman's blog.
I know it is just a suggestion, but had to try searching both, with no luck in getting the expected unexpected.
Earlier quoted context omitted.
Which software do you use to index the sites?
I wrote it myself from scratch. I have some metadata in mariadb, but the index is bespoke. A design sketch of the index is that it uses one file with sorted URL IDs, one with IDs of N-grams (i.e. words and word-pairs) referring to ranges in the URL file; as well as a dictionary for relating words to word-IDs; that's a GNU Trove hash map I modified to use memory map data instead of direct allocated arrays. So when you…
Earlier quoted context omitted.
How so? It's intuitive and super fast. I whish there were more websites with such a simple UI.
Some people like a flashy UI, the modern look is important for them. It's ok to have aesthetic preferences, let's not pretend we don't all have them. In the end, it's a niche search engine I've made, the intended audience is the long tail. It just isn't for everyone, and if it was for everyone, it probably would be lesser for it.
I'm developing a text-heavy site and philosophically I'm trying to view documents as just that... documents [1]. But I don't get good results for "rug pull". - 1 https://rugpullindex.com
Some rigmarole getting recognized as a good bot by the CDNs. I've submitted a request fairly recently, but haven't heard back from them yet.
Like I would like to be on good terms with them, and other websites that block small independent crawlers.
I can't blame them though, there's a lot of bad bots out there. But I'm doing my best not be part of the problem.
Earlier quoted context omitted.
How did you go about seeding your web crawler with URLs to crawl?
I just started with my website and did a crawl. Subsequently I've been seeding it with the best results form my previous crawls. It's a directed search so it doesn't seem to need a particularly solid seed to get decent results.
I'm developing a text-heavy site and philosophically I'm trying to view documents as just that... documents [1]. But I don't get good results for "rug pull". - 1 https://rugpullindex.com
I looked at the source for your site's front page. That's not text-heavy; that's markup-heavy. I didn't bother looking at the rest of the pages because it appears to be yet another crypto market site.
from About page: > If you search for "Plato", you might for example end up at the Canterbury Tales. Go looking for the Canterbury Tales, and you may stumble upon Neil Gaiman's blog. I know it is just a suggestion, but had to try searching both, with no luck in getting the expected unexpected.
Fascinating. I studied an "obscure" group of insects. My go-to search term to test an engine is their family name as it is a rarely used word and I know most (all?) of the major data sources that have accumulated data on it. When Wolfram Alpha added species names, I checked with the name, boring, Duck Duck, boring, Google (well we know Google isn't for search anymore, it's absolutely horrible) boring, Bing, boring...…