Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It seems that Google is forgetting the old web

stop.zona-m.net

Post reply on HN
Print view View original

61–70 of 311 posts

Re: It seems that Google is forgetting the old web

#61

Post by xorand »

eruci wrote: post #48
xorand wrote: post #27

Earlier quoted context omitted.

Yesterday I noticed that Google Scholar forgot one of my articles from 2018, on arXiv. See: https://scholar.google.com/scholar?q=arXiv%3A1811.04960 Google Scholar is not the same as Google Search, which can still find it https://www.google.com/search?q=arXiv%3A1811.04960 For how long, I have no idea. The article was at the same link all the time and arXiv is very reputable.

I also noticed that all our scholarly articles are gone from Google Scholar. The only thing there is our two highly cited books. https://scholar.google.ca/scholar?hl=en&as_sdt=0%2C5&q=site%... We've come to rely on Google too much, so much that if you are not on Google you don't exist. That's a problem with researchers that are looking for articles to cite.

Is this recent? In my case I noticed it yesterday.

Re: It seems that Google is forgetting the old web

#62

Post by Tolyo »

inetknght wrote: post #7

I have noticed that searching for exact quotes seems to have been broken on Google for a few years. But only minimally broken. And I've had no idea how to reason with it. This article completely corresponds with problems I've encountered with searching for results on StackOverflow or software documentation sites; it's especially perplexing that "site:..." combined with exact quotes does not work for many cases. Googl…

Ive noticed lately some of the search operators(intitle, inurl) have more limited results. Feels like a step backwards in functionality, but I suppose its a step forwards in getting the user on Amazon or whatever

Re: It seems that Google is forgetting the old web

#63

Post by eruci »

xorand wrote: post #61
eruci wrote: post #48

Earlier quoted context omitted.

I also noticed that all our scholarly articles are gone from Google Scholar. The only thing there is our two highly cited books. https://scholar.google.ca/scholar?hl=en&as_sdt=0%2C5&q=site%... We've come to rely on Google too much, so much that if you are not on Google you don't exist. That's a problem with researchers that are looking for articles to cite.

Is this recent? In my case I noticed it yesterday.

I also just noticed it. No idea when the rest of the papers were dropped.

Re: It seems that Google is forgetting the old web

#64

Post by bshipp »

puzzle wrote: post #50
0815test wrote: post #34

Earlier quoted context omitted.

The index only has to be on "low latency storage" if low-latency results to any query are required. While that's definitely true of the modal "Google Search", most of these queries for "long-tail, old content" as discussed in the OP don't really need that sort of quick response.

Interview questions: How would a search engine distinguish between the two kinds of queries, tens of thousands of times a second? And how would one architect such a two-tiered system, particularly with an eye toward cascading failures?

I don't work at Google so I'm probably way off base, but if I was designing it I wouldn't bother telling the difference between the two types of queries.

I'd break up the indices into digestible chunks, perhaps chronologically by year/month crawled, and then run all queries simultaneously (in parallel) against all those index chunks and combine the results at the end. Infinitely scalable and can be tweaked to ensure specific response times.

And there'd definitely be no need to set some arbitrary date cut-off; just add a few more virtual machines. I'd bet that's what Google was doing, and then scaled back those machines to save money and boost profits.

Re: It seems that Google is forgetting the old web

#66

Post by typon »

puzzle wrote: post #50
0815test wrote: post #34

Earlier quoted context omitted.

The index only has to be on "low latency storage" if low-latency results to any query are required. While that's definitely true of the modal "Google Search", most of these queries for "long-tail, old content" as discussed in the OP don't really need that sort of quick response.

Interview questions: How would a search engine distinguish between the two kinds of queries, tens of thousands of times a second? And how would one architect such a two-tiered system, particularly with an eye toward cascading failures?

Make it opt-in. Instead of requiring every search to be finished in less than 0.5 seconds, allow users to tick a box that says "Take your time" and pull indices from slow storage in that case. If I know I want something niche, I am willing to wait the extra few seconds or even a minute.

Re: It seems that Google is forgetting the old web

#67

Post by founderling »

vissi wrote: post #60
founderling wrote: post #53

Earlier quoted context omitted.

Here is an example: http://www.gnoosic.com/discussion/metallica__5.html No matter how you search for the content on Google, nothing comes up: https://www.google.com/search?q="Metallica+only+played+2+son... DuckDuckGo has it: https://duckduckgo.com/?q="Metallica+only+played+2+songs+fro... I checked the wayback machine and the content has constantly been on that url for over 10 years. This is the first example of an ol…

Well, search may have bugs (or undocumented features) too, I have googled content from other pages on this site (related to Metallica). Page 6 for example: https://www.google.com/search?q=%22Listen+up+you+fags+metall...

I don't think the articles premise is that Google axed all content older then 5 years or so. But that it gradually discards old unique content.

Which goes against the original mission of Google to "organize the world's information and make it universally accessible".

A "bug" could be an option, but I don't expect that to be the reason. It's too easy to find examples of forgotten content. And I don't think a bug of that magnitude in Googles core business would go unnoticed.

Re: It seems that Google is forgetting the old web

#68

Post by benologist »

kickscondor wrote: post #25
degenerate wrote: post #13

This author is in his own little bubble and doesn't understand the vast amount of blog-repost spam that google has to deal with. The way their algorithm most likely deals with this is a mixture of domain rank + tenure... how long has this copy of this article existed on this domain, and can we be sure this is the original copy? The author says the article was removed in 2006 (" [...] posts, were not accessible anymor…

Part of the problem is that their algorithm has become weighted against blogs and personal websites. > Rumors spread that large link pages (for surfing) might be considered “link farms” (and yes on SEO sites they were but these things eventually trickle down to little personal site webmasters too) so these started to be phased out. Then the worry was Blogrolls might be considered link farms so they slowly started to…

That's exactly what it is, and Google's also incentivizing many low-quality sites to engage disproportionately in SEO to boost their Google Adsense earnings too.

Re: It seems that Google is forgetting the old web

#69

Post by growt »

kickscondor wrote: post #17

While it's become impossible to browse the wider Web with Google, it's getting a bit easier elsewhere. A few helpful search engines: * https://millionshort.com/ * https://wiby.me/ * https://pinboard.in/search/ A recent movement to build personal Yahoo!-style directories: * https://href.cool/ (my own project) * https://indieseek.xyz/ * https://districts.neocities.org/ * https://the.dailywebthing.com/ The above resourc…

Looks like millionshort is already dead? Last and only blog post is over a year old. copyright is still 2018. Sad, it looked promising.

Re: It seems that Google is forgetting the old web

#70

Post by LeifCarrotson »

kickscondor wrote: post #25
degenerate wrote: post #13

This author is in his own little bubble and doesn't understand the vast amount of blog-repost spam that google has to deal with. The way their algorithm most likely deals with this is a mixture of domain rank + tenure... how long has this copy of this article existed on this domain, and can we be sure this is the original copy? The author says the article was removed in 2006 (" [...] posts, were not accessible anymor…

Part of the problem is that their algorithm has become weighted against blogs and personal websites. > Rumors spread that large link pages (for surfing) might be considered “link farms” (and yes on SEO sites they were but these things eventually trickle down to little personal site webmasters too) so these started to be phased out. Then the worry was Blogrolls might be considered link farms so they slowly started to…

You're jumping from describing observable results to a state of mind or motive which you can't observe.

> Then the worry was Blogrolls might be considered link farms so they slowly started to be phased out. Then the biggie: when Google deliberately filtered out all the free hosted sites from the SERP’s...

That's all observable fact.

Why? Because they were taking up space in the first 20 organic returns knocking out corporate and commercial sites and the sites likely to become paying customers were complaining.

I think the more reasonable, less diabolical motive was that the blogs and free hosted sites were largely link farms that no one wanted to visit.

It sucks for the few legitimate pages on those platforms, but when most of the legitimate page is the rare gem in a minefield of automated copies of other blogs, just with SEO links and ads inserted.

It's like a comments section: without moderation or captchas or both, a "thriving local community" on, say, a small town news site can be overwhelmed by automated pharmaceuticals spam. Then the newspaper kills the comment section, not out of any malice towards the original community but because they don't want to deal with the spam.

And yeah, dealing with spam and black hat SEO does take resources. If you (or worse, your chosen blog host) don't keep the weeds down, soon your pasture will be overrun and burned off.

Post reply on HN