41–50 of 311 posts
This author is in his own little bubble and doesn't understand the vast amount of blog-repost spam that google has to deal with. The way their algorithm most likely deals with this is a mixture of domain rank + tenure... how long has this copy of this article existed on this domain, and can we be sure this is the original copy? The author says the article was removed in 2006 (" [...] posts, were not accessible anymor…
This author is in his own little bubble and doesn't understand the vast amount of blog-repost spam that google has to deal with. The way their algorithm most likely deals with this is a mixture of domain rank + tenure... how long has this copy of this article existed on this domain, and can we be sure this is the original copy? The author says the article was removed in 2006 (" [...] posts, were not accessible anymor…
Earlier quoted context omitted.
Crawling isn't the real problem, nor is the bulk storage for the crawled pages. What do you do with these pages after you've crawled them? You need to build an index out of them, and serve that index out of some kind of low latency storage (DRAM, Flash). That makes increasing the index size very expensive. The index size has to be limited, and selecting the right pages to include in the index is thus a core quality f…
I'm having trouble imagining that Google would be more limited by the ratio of hardware power vs data size today than it was in the early days. If keeping the whole index in DRAM is now a requirement, then yes, I'd expect a hugely reduced overall dataset - but wouldn't that affect way more sites/pages than the comparatively few dropped historical records? I still suspect that this whole thing is more about bias (and…
[0] https://searchengineland.com/google-dumps-the-supplemental-r...
Earlier quoted context omitted.
Crawling isn't the real problem, nor is the bulk storage for the crawled pages. What do you do with these pages after you've crawled them? You need to build an index out of them, and serve that index out of some kind of low latency storage (DRAM, Flash). That makes increasing the index size very expensive. The index size has to be limited, and selecting the right pages to include in the index is thus a core quality f…
I'm having trouble imagining that Google would be more limited by the ratio of hardware power vs data size today than it was in the early days. If keeping the whole index in DRAM is now a requirement, then yes, I'd expect a hugely reduced overall dataset - but wouldn't that affect way more sites/pages than the comparatively few dropped historical records? I still suspect that this whole thing is more about bias (and…
It's actually more complicated than just a single static index, which is also why it's unrealistic to expect a search engine to be deterministic at scale.
Like it's sometimes feel like web completely frozen and all content moved into closed gardens. I switched to DDG a while ago for this and bunch of other reasons, but I wonder if someone else noticed this. Anyone?
While it's become impossible to browse the wider Web with Google, it's getting a bit easier elsewhere. A few helpful search engines: * https://millionshort.com/ * https://wiby.me/ * https://pinboard.in/search/ A recent movement to build personal Yahoo!-style directories: * https://href.cool/ (my own project) * https://indieseek.xyz/ * https://districts.neocities.org/ * https://the.dailywebthing.com/ The above resourc…
In my opinion there has to be widespread fatigue of Google just somehow managing to return a large chunk of something like 1,000 sites for pretty much any search. It's in part SEO, but it's also like the article mentions - Google makes money from ads. These sites they spam at you generate substantial revenue for Google - no name sites do not. Being the world's largest advertising corporation and search engine is one hell of a conflict of interest in terms of delivering what the user wants, instead of delivering what Google wants.
Google Scholar started to forget articles https://news.ycombinator.com/item?id=19599365
Downvoted against the evidence given. Anybody cares to explain why? Thank you.
It happens far less frequently than in other places of the web, but I've seen it happen often enough with some of my comments.
This author is in his own little bubble and doesn't understand the vast amount of blog-repost spam that google has to deal with. The way their algorithm most likely deals with this is a mixture of domain rank + tenure... how long has this copy of this article existed on this domain, and can we be sure this is the original copy? The author says the article was removed in 2006 (" [...] posts, were not accessible anymor…
Yesterday I noticed that Google Scholar forgot one of my articles from 2018, on arXiv. See: https://scholar.google.com/scholar?q=arXiv%3A1811.04960 Google Scholar is not the same as Google Search, which can still find it https://www.google.com/search?q=arXiv%3A1811.04960 For how long, I have no idea. The article was at the same link all the time and arXiv is very reputable.
We've come to rely on Google too much, so much that if you are not on Google you don't exist. That's a problem with researchers that are looking for articles to cite.
While it's become impossible to browse the wider Web with Google, it's getting a bit easier elsewhere. A few helpful search engines: * https://millionshort.com/ * https://wiby.me/ * https://pinboard.in/search/ A recent movement to build personal Yahoo!-style directories: * https://href.cool/ (my own project) * https://indieseek.xyz/ * https://districts.neocities.org/ * https://the.dailywebthing.com/ The above resourc…
Earlier quoted context omitted.
Crawling isn't the real problem, nor is the bulk storage for the crawled pages. What do you do with these pages after you've crawled them? You need to build an index out of them, and serve that index out of some kind of low latency storage (DRAM, Flash). That makes increasing the index size very expensive. The index size has to be limited, and selecting the right pages to include in the index is thus a core quality f…
The index only has to be on "low latency storage" if low-latency results to any query are required. While that's definitely true of the modal "Google Search", most of these queries for "long-tail, old content" as discussed in the OP don't really need that sort of quick response.
How would a search engine distinguish between the two kinds of queries, tens of thousands of times a second?
And how would one architect such a two-tiered system, particularly with an eye toward cascading failures?