Ask HN: Why does Pinterest dominate Google text search results? • Hacker News Board

Re: Ask HN: Why does Pinterest dominate Google text search results?

#101

Post by Someone » Fri, Dec 25, 2020, 9:33 PM UTC

lopmotr wrote: post #89

knuckleheads wrote: post #34

Earlier quoted context omitted.

> What incentive would Google have to continue populating that index? Presumably they would still want to run google.com and make money off of it. > Would I be breaking the law if I independently crawled and hosted an index without publishing an API for it? No. You would not get the advantage that Google gets when it crawls the web and so would not have access to a large amount of data that nobody else has access to.…

Do any site operators actually block non-Google search engine crawlers because being listed DDG/Bing/etc isn't worth the extra cost of serving the crawler? It sound a bit ridiculous unless they actually don't want to be found. Maybe they only allow GoogleBot because that's all they thought of and the extra cost is in researching what all the other search engines call theirs. Perhaps other search engines should spoof…

I would think the site owner’s cost of being indexed is the same for every search engine that indexes the site.

The benefit varies with the quality of the search engines, and that will vary between search engines, but it does get larger the more a search engine is used, so a cost/benefits analysis may show Google and a few other large ones are the only ones worth supporting.

Re: Ask HN: Why does Pinterest dominate Google text search results?

#102

Post by loeg » Fri, Dec 25, 2020, 9:34 PM UTC

lopmotr wrote: post #89

knuckleheads wrote: post #34

Earlier quoted context omitted.

> What incentive would Google have to continue populating that index? Presumably they would still want to run google.com and make money off of it. > Would I be breaking the law if I independently crawled and hosted an index without publishing an API for it? No. You would not get the advantage that Google gets when it crawls the web and so would not have access to a large amount of data that nobody else has access to.…

Do any site operators actually block non-Google search engine crawlers because being listed DDG/Bing/etc isn't worth the extra cost of serving the crawler? It sound a bit ridiculous unless they actually don't want to be found. Maybe they only allow GoogleBot because that's all they thought of and the extra cost is in researching what all the other search engines call theirs. Perhaps other search engines should spoof…

Yes, site operators actually block non-Googlebot crawlers. See the example [0] from https://news.ycombinator.com/item?id=25538842 .

Spoofing crawler identity completely defeats the point of the honor-system robots.txt.

Re: Ask HN: Why does Pinterest dominate Google text search results?

#103

Post by grishka » Fri, Dec 25, 2020, 9:35 PM UTC

knuckleheads wrote: post #95

grishka wrote: post #92

Earlier quoted context omitted.

So, uh, don't respect robots.txt in your search engine? It's not like there's a law that you have to, and that you can't pretend you are Googlebot. The only real obstacle I can imagine is that some firewalls might be configured to be more permissive with traffic originating from Google subnets.

You would be blocked fairly quickly by many website operators and no longer able to access those websites if you straight up ignored robots.txt files. You also might even end up being served cease and desists by some websites and sued if you continue to persist and try to find ways around it.

And what if you do respect it but follow Googlebot rules?

Re: Ask HN: Why does Pinterest dominate Google text search results?

#104

Post by knuckleheads » Fri, Dec 25, 2020, 9:37 PM UTC

lopmotr wrote: post #89

knuckleheads wrote: post #34

Earlier quoted context omitted.

> What incentive would Google have to continue populating that index? Presumably they would still want to run google.com and make money off of it. > Would I be breaking the law if I independently crawled and hosted an index without publishing an API for it? No. You would not get the advantage that Google gets when it crawls the web and so would not have access to a large amount of data that nobody else has access to.…

Do any site operators actually block non-Google search engine crawlers because being listed DDG/Bing/etc isn't worth the extra cost of serving the crawler? It sound a bit ridiculous unless they actually don't want to be found. Maybe they only allow GoogleBot because that's all they thought of and the extra cost is in researching what all the other search engines call theirs. Perhaps other search engines should spoof…

> Do any site operators actually block non-Google search engine crawlers because being listed DDG/Bing/etc isn't worth the extra cost of serving the crawler?

Many website operators do actually block crawlers from non Google search engines and it's because the cost of being crawled isn't worth it to them. Here's a good quote from one such webmaster:

    As a webmaster I get a bit tired of constantly having to deal with the startup crawler du jour.

    From law firms looking for DMCA violations to verticals search engines, to image aggregators, to company intelligence resellers… It feels to me that everybody and their brother has gotten into spidering sites.

    With 10,000s of pages that have content that is only relevant to a targeted audience who is perfectly able to find us on the majors, I do not hesitate to block (and possibly ban) when I see an aggressive crawler that does not provide me or my customers with direct benefits.

Taken from http://www.skrenta.com/2008/04/cuill_is_banned_on_10000_site...

> Perhaps other search engines should spoof GoogleBot. Browsers have being doing that since forever spoofing Netscape (Mozilla), Safari, etc. for the same reason.

People have tried this and it doesn't work. Google provides ways to check to make sure traffic is coming from Google IP addresses and practitioners and academics study how to spot fake Googlebots. https://developers.google.com/search/docs/advanced/crawling/... https://blogs.akamai.com/2014/07/search-engine-impersonation... https://ieeexplore.ieee.org/document/8421894

> This sounds like a common fallacy of people criticizing the free market.

I am asserting that crawling the web is a natural monopoly. This means that the free market has failed and that it is not possible for the market to heal itself in this regard. There is significant evidence that this is the case and I imagine you'll be hearing more and more about it soon.

Re: Ask HN: Why does Pinterest dominate Google text search results?

#105

Post by moultano » Fri, Dec 25, 2020, 9:38 PM UTC

KKKKkkkk1 wrote: post #61

blunte wrote: post #13

Earlier quoted context omitted.

Google should be asking from a product perspective if they are providing the search results the user is most likely seeking. I seriously doubt that most pinterest results are what users are seeking. As such, Google probably should consider changes to their algorithm to compensate (cat and mouse) for pinterest's gaming. That is assuming how pinterest does it is not against any google rules... because if it were agains…

With Google featuring progressively more ads and Google content at the top of the results page, it's not clear if it's still in Google's self interest to give users the best results anymore.

This is why there's an aggressive firewall between the ads and search organizations. The ranking team isn't even aware of revenue numbers by ACL.

Re: Ask HN: Why does Pinterest dominate Google text search results?

#106

Post by intrasight » Fri, Dec 25, 2020, 9:39 PM UTC

It does dominate image search which sucks. I instinctively add "-pinterest" to my search. I don't have this problem on text search.

Re: Ask HN: Why does Pinterest dominate Google text search results?

#107

Post by knuckleheads » Fri, Dec 25, 2020, 9:40 PM UTC

grishka wrote: post #103

knuckleheads wrote: post #95

Earlier quoted context omitted.

You would be blocked fairly quickly by many website operators and no longer able to access those websites if you straight up ignored robots.txt files. You also might even end up being served cease and desists by some websites and sued if you continue to persist and try to find ways around it.

And what if you do respect it but follow Googlebot rules?

Applebot was able to get away with doing exactly this but I imagine that's because it's Apple and websites knew that Apple was about to send them enough traffic via Apple News to make it worth their while. I don't know if other search engine operators have tried this but I would imagine they would get caught by rate limiters set for non Google IP's and then they would be blocked.

Re: Ask HN: Why does Pinterest dominate Google text search results?

#108

Post by knuckleheads » Fri, Dec 25, 2020, 9:41 PM UTC

Someone wrote: post #101

lopmotr wrote: post #89

Earlier quoted context omitted.

Do any site operators actually block non-Google search engine crawlers because being listed DDG/Bing/etc isn't worth the extra cost of serving the crawler? It sound a bit ridiculous unless they actually don't want to be found. Maybe they only allow GoogleBot because that's all they thought of and the extra cost is in researching what all the other search engines call theirs. Perhaps other search engines should spoof…

I would think the site owner’s cost of being indexed is the same for every search engine that indexes the site. The benefit varies with the quality of the search engines, and that will vary between search engines, but it does get larger the more a search engine is used, so a cost/benefits analysis may show Google and a few other large ones are the only ones worth supporting.

Yes! Exactly!

Re: Ask HN: Why does Pinterest dominate Google text search results?

#109

Post by tomaszs » Fri, Dec 25, 2020, 9:41 PM UTC

I read the question as follows: how to rank so high like Pinterest does?

The answer is: you should not spend any second on SEO. It is time and money wasted if you are a SMB. Anything else, including taking a walk is better for your business than going into SEO optimization.

Your clients are people who buy your stuff. Search engine robot is not your client. After years of working with SEO it becomes clear for me, if you satisfy a robot, you don't satisfy your client. These are two different things.

The bright side is that when you will serve your clients, robots will catch up eventually. But it is search engine problem not something you should spend time on.

There is no magic bullet method to rank high. No special hack you can use. Everything you will read about SEO is smokes and mirrors. If you have million dollar company and did everything else right - sure, you can throw hundreds thousands dollars into SEO and pray search engine devs won't change their minds.

But as a SMB company, counting each dollar, just leave it. You won't win over search engines and people with bulk of money.

Spend your time and effort on clients. It is the only way to spend it right. If search engine will change mind - so be it. You won't be dependant on it. And your clients will come to you because they love your product, it is stronger than being number one in a twisted search ranking.

Re: Ask HN: Why does Pinterest dominate Google text search results?

#110

Post by monkeybutton » Fri, Dec 25, 2020, 9:42 PM UTC

mrtksn wrote: post #56

blunte wrote: post #13

Earlier quoted context omitted.

Google should be asking from a product perspective if they are providing the search results the user is most likely seeking. I seriously doubt that most pinterest results are what users are seeking. As such, Google probably should consider changes to their algorithm to compensate (cat and mouse) for pinterest's gaming. That is assuming how pinterest does it is not against any google rules... because if it were agains…

>Google should be asking from a product perspective if they are providing the search results the user is most likely seeking. I am cynical. Maybe Google's own experiments show that users click more ads when they get frustrated of the Pinterest spam that is technically not spam enough to be removed from the index. What the users are going to do? Use Bing? Google's most optimised version is probably being an Ads search…

It's certainly going in that direction. With reverse image search being replaced with "Search with Google Lense" on mobile, the most prominent results are now all Google shopping ads.