Yep. And I thought I was the only one thinking Google's brainwashed web isn't representative of what's out there. It's not only the old stuff, though. I have a suspicion Google won't show you pages without Google ads on them.
It seems that Google is forgetting the old web
11–20 of 311 posts
Re: It seems that Google is forgetting the old web
#12The assertion that this is because "indexing the whole Web is crushingly expensive, and getting more so every day" is a bit flawed. Since old content is very unlikely to be updated, it doesn't have to be re-crawled a lot. I'm certain Google has a score that tells it how often the content of a given site is likely to change. This argument of expense becomes even less durable when you consider that DuckDuckGo, a company with an infinitesimal fraction of Google's resources, is perfectly able to keep that kind of content in its database.
I agree with the observation that this is about shifting everything to current data, because people overwhelmingly care about things that happened a few days ago. There used to be a long tail of users searching for old data and references, but I suspect they're fading away. Biasing the index towards recency also has legal advantages for Google, because delisting old content makes it less likely to receive takedown requests in connection with "right to be forgotten" legislation.
Re: It seems that Google is forgetting the old web
#13This author is in his own little bubble and doesn't understand the vast amount of blog-repost spam that google has to deal with. The way their algorithm most likely deals with this is a mixture of domain rank + tenure... how long has this copy of this article existed on this domain, and can we be sure this is the original copy?
The author says the article was removed in 2006 ("[...] posts, were not accessible anymore") and then he re-posted the article at a new domain in 2013. That means any copy/crawl/repost of the article from 2006-2012 is now the oldest living, and thus "original", version of the article. His 2013 repost was seen as just another blog-spam copy.
Google is not forgetting the old web unless we see evidence of content disappearing from the index that have been consistently hosted at the same domain & URL since their original posts. Unless you properly 301 your URLs to new locations and consistently host your content, it's a guessing game for the crawler to determine where the original content has moved to.
Re: It seems that Google is forgetting the old web
#14I have noticed that searching for exact quotes seems to have been broken on Google for a few years. But only minimally broken. And I've had no idea how to reason with it. This article completely corresponds with problems I've encountered with searching for results on StackOverflow or software documentation sites; it's especially perplexing that "site:..." combined with exact quotes does not work for many cases. Googl…
> Google certainly doesn't seem to value feedback at all.
As it is with most big companies that profit from ad revenue. They seem to consider performance indicators to be sufficient to know if a new feature is good or bad, instead of worrying about written customer feedback.
Re: It seems that Google is forgetting the old web
#15I maintain two web sites that date back to 2003. They are still very active (thousands to tens of thousands of uniques per day), but I and my users have noticed that only the more recent content (2012+) shows up in Google.
In a way I’m had to hear that Google is delisting the older content because I thought I was doing something wrong.
But it’s still frustrating for my visitors because every few months I get a message about how they can’t believe all the information there is on the site that they’ve searched for for years but never found through search engines but it’s all right there on this one site. (It’s something of a regional history site.)
I guess those sites have involuntarily become part of the “dark web.”
Re: It seems that Google is forgetting the old web
#16When web directories like Yahoo lost out to web search engines like Google, we lost something crucial. While search is good for answering questions you know how to ask, browsing was exploratory and led us to know what we didn't know. When it comes to learning a complex topic like mathematics, this kind of serendipity was very useful. There are some amazing resources on the web, but googling won't let you discover those.
Sometimes, the same idea is available in a book, in a TED talk, and in a podcast. Some of us are curating such resources categorized by topic / format / year / difficulty / estimated time. Our GitHub repo received 100+ stars in less than a week, so I thought it would be a good time to show it to HN. I'd love to get some feedback and critique from the HN community where I have learned and discovered so much.
Here's the Show HN post: https://news.ycombinator.com/item?id=19604295
Re: It seems that Google is forgetting the old web
#17While it's become impossible to browse the wider Web with Google, it's getting a bit easier elsewhere.
A few helpful search engines:
A recent movement to build personal Yahoo!-style directories:
* https://href.cool/ (my own project)
* https://districts.neocities.org/
* https://the.dailywebthing.com/
The above resources are focused on general blogging and personal websites - for software and startups, I would refer to the appropriate 'awesome' directories. (https://github.com/sindresorhus/awesome or https://awesomelists.top)
If you know of any more, please list them - a small group of us are collecting these and trying to encourage new projects.
Re: It seems that Google is forgetting the old web
#18The post unflatteringly compares Google with DuckDuckGo. But doesn't DuckDuckGo use Google's technology?
Re: It seems that Google is forgetting the old web
#19The assertion that this is because " indexing the whole Web is crushingly expensive, and getting more so every day " is a bit flawed. Since old content is very unlikely to be updated, it doesn't have to be re-crawled a lot. I'm certain Google has a score that tells it how often the content of a given site is likely to change. This argument of expense becomes even less durable when you consider that DuckDuckGo, a comp…
Yeah, it seems a natural consequence of the combination of vast amounts of recent content with the fact that people mostly want recent content. To pick one trivial example from yesterday, if I'm looking for help with an interface issue with some current version of a program, forum posts from 10 years ago are probably not useful.
Information that people regularly access for whatever reason will tend to remain relatively visible. But, yeah, relatively obscure older content is just going to get drowned out unless you know exactly where and how to look. One might argue with Google's criteria around relevance. However, that older information is going to get harder and harder to find just in the natural course of things.
Re: It seems that Google is forgetting the old web
#20This author is in his own little bubble and doesn't understand the vast amount of blog-repost spam that google has to deal with. The way their algorithm most likely deals with this is a mixture of domain rank + tenure... how long has this copy of this article existed on this domain, and can we be sure this is the original copy? The author says the article was removed in 2006 (" [...] posts, were not accessible anymor…
The first paragraph of the article mention the story of Tim Bray[0], which is exactly about this : Google forgetting an article which did not change location.
[0] : https://www.tbray.org/ongoing/When/201x/2018/01/15/Google-is...