Live data from Hacker News

CNET is deleting old articles to try to improve its Google Search ranking

theverge.com

81–90 of 592 posts

Re: CNET is deleting old articles to try to improve its Google Search ranking

#81

Earlier quoted context omitted.

The theory I've heard is related to 'crawl budget'. Google is only going to devote a finite amount of time to indexing your site. If the number of articles on your site exceeds that time, some portion of your site won't be indexed. So by 'pruning' undesirable pages, you might boost attention on the articles you want indexed. No clue how this ends up working in practice. Google's suggestion isn't to delete pages, but…

Even if that rule were true, why wouldn’t everything in the say, top NNN internet sites get an exemption? It is the Internet’s most hit content, why would it not be exhaustively indexed? Alternatively, other than ads, what is changing on a CNN article from 10 years ago? Why would that still be getting daily scans?

Probably bad technology detecting a change. Things like current news showing up beneath the article, which changes whenever a new article is added. I've seen this happen on quite a few large websites. It might be technologically easier to drop old articles than the amount of time to fix whatever they use to determine if a page has changed. You would think a site like CNET wouldn't have to deal with something like that, but sometimes these sites that have been around for a long time have some serious outdated tech.

Re: CNET is deleting old articles to try to improve its Google Search ranking

#82

Earlier quoted context omitted.

The theory I've heard is related to 'crawl budget'. Google is only going to devote a finite amount of time to indexing your site. If the number of articles on your site exceeds that time, some portion of your site won't be indexed. So by 'pruning' undesirable pages, you might boost attention on the articles you want indexed. No clue how this ends up working in practice. Google's suggestion isn't to delete pages, but…

But as that linked guide explains, that's only relevant for sites with e.g. over a million pages changing once a week . That's for stuff like large e-commerce sites with constantly changing product info. Google is clear that if your content doesn't change often (in the way that news articles don't), then crawl budget is irrelevant.

To determine whether content changes Google has to spend budget as well, hasn't it? So it has to fetch that 20-years old article.

Re: CNET is deleting old articles to try to improve its Google Search ranking

#83
post #40

Earlier quoted context omitted.

In fairness to them. If one of their top ways of getting traffic is being marked as less relevant because of older articles, what do you want them to do? Just continue to lose money because Google can't rank them appropriately?

Or mayhap Google et al could realize and understand that having old articles in archive shouldn't penalize your ranking.

Yep. Also, the days of the "Advanced Search" page have passed, but Google still has time range options under the "Tools" button near the top of the results page. If they're giving the user the option to filter results by time, then it's pretty goofy for the algorithm to de-rank a site in results where the the default of "Any time" was selected in the query just because the site has old articles in the index.

Re: CNET is deleting old articles to try to improve its Google Search ranking

#84

Earlier quoted context omitted.

The theory I've heard is related to 'crawl budget'. Google is only going to devote a finite amount of time to indexing your site. If the number of articles on your site exceeds that time, some portion of your site won't be indexed. So by 'pruning' undesirable pages, you might boost attention on the articles you want indexed. No clue how this ends up working in practice. Google's suggestion isn't to delete pages, but…

They could specify in the sitemap how often do old articles change. Or set a indefinite caching header.

Google might not trust the sitemap because it sometimes is wrong.

Re: CNET is deleting old articles to try to improve its Google Search ranking

#85

Is there any evidence this would even work? Surely Google determines "fresh, relevant" content according to whatever has recently been published, which this doesn't change. If anything, doesn't Google consider sites with a long history of content with tons of inbound links as more authoritative and therefore higher-ranked? This baffles me. It baffles me why this would be successful SEO -- and assuming that it actuall…

The theory I've heard is related to 'crawl budget'. Google is only going to devote a finite amount of time to indexing your site. If the number of articles on your site exceeds that time, some portion of your site won't be indexed. So by 'pruning' undesirable pages, you might boost attention on the articles you want indexed. No clue how this ends up working in practice. Google's suggestion isn't to delete pages, but…

> The theory I've heard is related to 'crawl budget'. Google is only going to devote a finite amount of time to indexing your site.

Once a site has been indexed once, should it really be crawled again? Perhaps Google should search for RSS/Atom feeds on sites and poll those regularly for updates: that way they don't waste time doing to a site scrape multiple times.

Old(er) articles, once crawled, don't really have to be babysat. If Google wants to double-check that an already-crawled site hasn't changed too much, they can do a statistical sampling of random links on it using ETag / If-Modified-Since / whatever.

Re: CNET is deleting old articles to try to improve its Google Search ranking

#87

Is there any evidence this would even work? Surely Google determines "fresh, relevant" content according to whatever has recently been published, which this doesn't change. If anything, doesn't Google consider sites with a long history of content with tons of inbound links as more authoritative and therefore higher-ranked? This baffles me. It baffles me why this would be successful SEO -- and assuming that it actuall…

I have noticed some articles (and not just "Best XXX of 202Y" articles) that seem to always update their "Updated on" date which Google unhelpfully picks up and shows in search results leading me to think the page is much more recent than it is.

[deleted]

Re: CNET is deleting old articles to try to improve its Google Search ranking

#88
post #75

So google's shitty search now economically incentivizes sites to destroy information. Can there be any doubt that Google destroyed the old internet by becoming a bad search engine? Could their exclusion of most of the web be considered punishment for being sites being so old and stable that they don't rely on Google for ad revenue?

Google is destroying the internet is a good way to put it. AD dollars are their only priority. I hope Google dies because of it.

[deleted]

Re: CNET is deleting old articles to try to improve its Google Search ranking

#89

Earlier quoted context omitted.

But as that linked guide explains, that's only relevant for sites with e.g. over a million pages changing once a week . That's for stuff like large e-commerce sites with constantly changing product info. Google is clear that if your content doesn't change often (in the way that news articles don't), then crawl budget is irrelevant.

To determine whether content changes Google has to spend budget as well, hasn't it? So it has to fetch that 20-years old article.

> So it has to fetch that 20-years old article.

It doesn't have to fetch every article (statical sampling can give confidence intervals), and it doesn't have to fetch the full article: doing a "HEAD /" instead of a "GET /" will save on bandwidth, and throwing in ETag / If-Modified-Since / whatever headers can get the status of an article (200 versus 304 response) without bother with the full fetch.

Re: CNET is deleting old articles to try to improve its Google Search ranking

#90
post #75

So google's shitty search now economically incentivizes sites to destroy information. Can there be any doubt that Google destroyed the old internet by becoming a bad search engine? Could their exclusion of most of the web be considered punishment for being sites being so old and stable that they don't rely on Google for ad revenue?

Google is destroying the internet is a good way to put it. AD dollars are their only priority. I hope Google dies because of it.

What a fantasy. It does not show any sign of profit decrease. How would a company die with $279.8B revenue, steadily increasing yearly?
Post reply on HN