371–380 of 383 posts
Earlier quoted context omitted.
We should probably classify the crawler identifying problem as impossible and move along. Less resources wasted and easier automation for everyone. Assuming a crawler is malicious is narrow-minded.
https://developers.google.com/search/docs/advanced/verifying...
Earlier quoted context omitted.
> Harnessing real user experiences (blogs, etc) This is what we need more than anything. More independent blogs. The ability to search events now, or 10 years ago, mass indexing of RSS feeds, etc. A general search engine is kinda way out of the ballpark for now. But you could specialize for long form blogs, from all sides, hard-left, hard-right, women in tech, white supremacists, all the extremes and moderates. I've…
> More independent blogs. The problem is that content farms have mastered the art of writing like an ostensibly independent blog. This is most visible in recipe blogs, where for example the site will look independent, the blog owner’s "About Me" page will say that she is a young woman born and raised in Louisiana and passionate about her home region’s cooking, but the English is replete with the sort of mistakes that…
Earlier quoted context omitted.
How are you going to identify what's blogspam and what's legitimate without indexing it all in the first place?
You load, detect and discard it? We’re already talking about building a search engine, might as well talk about a model to convincingly detect blogspam too.
Earlier quoted context omitted.
> You do that by having a list of known good sites to answer specific types of questions, and looking at the sites they link to. I mean, that's basically the core of original Google pagerank, right? A "good" site linking to another site is what makes that other site some amount of "good" too, links from better sites carry more 'juice'. "good" is of course not just binary, but a quantitative weight. I don't know to wh…
That was exactly my thought on reading that part of the article: that the author was describing Pagerank, but substituting Pagerank's design objective of algorithmically outsourcing quality judgments to the global community of website publishers with the programmer's prejudices instead. I hope the project of improving on DuckDuckGo is successful though, and some of the other proposals in the article sound promising t…
> Instead, it should crawl a whitelist of domains, or “tier 1” domains. These would be the limited mainly to authoritative or high-quality sources for their respective specializations, and would be weighed upwards in search results. Not a big fan of this conclusion. Who chooses the white list, and why should I trust them? Is it democratically chosen? Just because a site is popular very clear does not mean it's trustw…
It uses Gigablast which has a much more fair search result set more akin to search engines of the past!
Check out the serious difficulties the Common Crawl had with crawling 1% of the public internet on donated money and then get back to me with a plan. This is really, really hard to do for free. Maybe talk to Gates :)
TFA specifically mentions not crawling the web, but using a curated list.
is somebody aware of a project where the end-user Browser acts as a Crawler? it already spent the energy to render the content. Readability.js extracts page section, does some processing for keywords, hashes anchor links, signs it and sends it off. Cache-Control response headers indicate if the page is public or private. Of course, where it is sending to will have an electricity bill to pay to index the submissions.
Yes. Both PeARS[1] and Cliqz[2] tried to do that. Both got direct support from Mozilla[3][4] but it looks like neither really kicked off. PeARS was meant to be installed voluntarily by users who would then choose to share their indexes only to those they personally trusted, so the idea is very privacy conscious but also very hard to scale. Cliqz, on the other hand, apparently tried to work around that issue by having…
I was aware Mozilla had some involvement with Cliqz, but didn't really pay attention, I remembered the company became owner of the Firefox Addon Ghostery some years ago. They closed shop mid 2020, but their tech-blog 0x65.dev is still up. There are a lot of posts from last December that explain its inner workings.
User-agents really do contribute their history, containing which search terms led to which pages. From this data (they named it the "human web") the search engine had a page model of which search terms led in higher frequency to the page. Related search terms were normalized. Only later did a "fetcher" really index high frequency content and consider it again in a later stage of search. Interesting bootstrap, more energy efficient maybe as it can run on less information.
Sending the search and browsing history offsite needs explaining and trust. But ultimately, any centralized search engine will see the search data too. Cliqz approach was trying to piggy-back on the result sets on search terms by established search engines, the search term + choosen result combination a donation of the user. Not any less invasive then other search engines. Would I send off the whole corpus of my browsing history? this raises good questions. Thanks for the links!
Earlier quoted context omitted.
I really wish Google would prioritize English results for English searches consistently. I'm living in Japan as a native English speaker, and have my OS, browser and logged in Google account all configured for English only. Despite that, Google search results always prioritize Japanese language content. Every now and then (though not consistently) it gives me a yellow popup asking if I'd like English results instead,…
There was a time, a long time ago, where google had this: www.google.com/ncr 'ncr' here stands for no country recognition. It allowed many expats to do technical searches without the noise of regionalization results. Of course someone clever at google figured out that was probably too useful and now it just redirects you back to google.com because screw all those niche use-cases.
When you were in a different country (e.g., India), and you typed in google.com out of habit, it would recognize your IP-geo and redirect you to the country-specific domain (e.g., google.co.in).
If you really just wanted google.com for whatever reason, then you'd type google.com/ncr. It then wouldn't redirect you based on your IP-geo, and you'd stay on google.com.
In other words, google.com/ncr _always_ redirected you back to google.com. Then, and now.