Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Only Google is really allowed to crawl the web

knuckleheads.club

Post reply on HN
Print view View original

111–120 of 365 posts

Re: Only Google is really allowed to crawl the web

#111

Post by supernovae »

marshmallow_12 wrote: post #55
throwaway_kufu wrote: post #27

Earlier quoted context omitted.

They are not just taking away internet traffic, but in the flights example, they actually acquired an aggregate flight/travel company and so they are actually entering markets and competing with their own ad customers. Then it comes fully circle to Google unfairly using their market position vis-a-vis data, search and advertising. It’s a win-win Google lets the data dictate which markets to enter and on one hand they…

Aren't there anti trust laws to prevent this kind of thing?

Just tell people to stop using google. Go direct.

Re: Only Google is really allowed to crawl the web

#112

Post by aetherane »

danielscrubs wrote: post #80
throwaway_kufu wrote: post #27

Earlier quoted context omitted.

They are not just taking away internet traffic, but in the flights example, they actually acquired an aggregate flight/travel company and so they are actually entering markets and competing with their own ad customers. Then it comes fully circle to Google unfairly using their market position vis-a-vis data, search and advertising. It’s a win-win Google lets the data dictate which markets to enter and on one hand they…

Be careful when using Google Flight, last time I checked they use significantly less margins between flights so it’s shorter trips but much riskier.

You can screwed any time you book a connecting flight on two different airlines even if the times aren't tight. For instance if one is cancelled.

If you use the same airline they will make sure you get to the destination.

Re: Only Google is really allowed to crawl the web

#113

Post by Mauricebranagh »

tyingq wrote: post #75
veltas wrote: post #63

Earlier quoted context omitted.

I swear something like 50% of those digests are totally incorrect as well. It's amazing they have kept the feature because it has never had a very high signal-to-noise ratio. I never trust what's presented in these digests without double-checking the source page.

I remember when rich snippets (one type of those widgets) came out there were a lot of funny examples. One for a common query about cancer treatments that pulled data from a dodgy holistic site saying that "carrots cured most types of cancer" (or something like that). There was a similar one where Google emphatically claimed a US quarter was worth five cents in a pretty and large snippet graphic.

I recall in the last uk election google got the infographic of party leaders about 60-70% wrong.

And quite often a people also ask refine is just some random guys comment from redit.

Re: Only Google is really allowed to crawl the web

#114

Post by lisper »

sesuximo wrote: post #64
lisper wrote: post #54

Earlier quoted context omitted.

I don't think you're quite clear on what the words "public" and "private" mean. "Public" is not a synonym for "run by the government" and "private" is not a synonym for "closed to everyone but the owner". Restaurants, for example, are generally open to the public, but they are not public. A restaurant owner is, with a few exceptions, free to refuse service to anyone at any time. If it's "exactly the same as a public…

Ok I am not clear then, but I’m less clear after your comment! In a public cache, who would you want to decide who has access? Is simply saying “anyone who pays has access” enough to qualify as public? if so, then I agree and this was my (possibly poorly phrased) intention in the original comment. But imo the restaurant model is also fine; in most cases people have access and it works.

> Is simply saying “anyone who pays has access” enough to qualify as public?

No because someone has to set the price, which is just an indirect method of controlling who has access.

> the restaurant model is also fine

It works for restaurants because there is competition. The whole point here is that web crawling/caching is a natural monopoly.

A better analogy here would be Apple's app store, or copyrighted standards with the force of law [1]. These nominally follow the "anyone who pays has access" model but they are not public, and the result is the same set of problems.

[1] https://www.thebrandprotectionblog.com/public-laws-private-s...

Re: Only Google is really allowed to crawl the web

#115

Post by kmeisthax »

mrweasel wrote: post #26

While I don’t disagree with the idea that all crawlers should have equal access, we also need to address the quality of many crawlers. Google and Microsoft have never hammered any website I’ve run into the ground. Crawlers from other other, smaller, search engines have, to the point where it was easier to just block them entirely. Part of the problem is that sites want search engine to index their site, but not allow…

I think this is a problem which should be solved by automatic rate-limiting and throttling at the application/caching layer (or just individual web server for smaller sites). Requests with a non-browser UA get put into a separate bots-only queue that drains at a rate of ~1/sec or so. If the queue fills up you start sending 429s with random early failures for bots (UA/IP/subnet pairs) that are overrepresented in the traffic flow.

I don't know if such software exists, but it should. It would be a hell of a lot healthier for the web than "everyone but Google f*ck off", and it creates an incentive for bots to throttle themselves (as they're more likely to get a faster response than trying to request as fast as possible).

Re: Only Google is really allowed to crawl the web

#116

Post by supert56 »

tyingq wrote: post #12

The bigger problem, to me, is not around crawling. It's the asymmetrical power Google has after crawling. Google is obviously on a mission to keep people on Google owned properties. So, they take what they crawl and find a way to present that to the end user without anyone needing to visit the place that data came from. Airlines are a good example. If you search for flight status for a particular flight, Google prese…

Perhaps I am misunderstanding or over simplifying things but it always surprises me that there are legal cases brought against companies who scrape data when so many of Google's products are doing exactly this.

It definitely feels like one set of rules for them and a different set for everyone else.

Re: Only Google is really allowed to crawl the web

#117

Post by dillondoyle »

suicas wrote: post #18
henriquez wrote: post #2

I'd like to see some data on their claim that website operators are giving googlebot special privileges. As far as I can tell it would be a huge pain in the ass to block crawler bots from my servers, not that I've tried. I have some weird pages that tend to get crawlers caught in infinite loops, and I try to give them hints with robots.txt but most of the bots don't even respect robots.txt. If I actually wanted to re…

A company I worked for ~7 years ago ran its own focused web crawler (fetching ~10-100m pages per month, targeting certain sections of the web). There were a surprising number of sites out there that explicitly blocked access to anyone but Google/Bing at the time. We'd also get a dozen complaints or so a month from sites we'd crawled. Mostly upset about us using up their bandwidth, and telling us that only Google was…

Isn't that the website owners right though? I'm not sure I understand the problem here.

If Google is taking traffic and reducing revenue, a company can deny in robots.txt. Google will actually follow those rules - unlike most others that are supposedly in this 2nd class.

Re: Only Google is really allowed to crawl the web

#119

Post by visarga »

sesuximo wrote: post #25
lisper wrote: post #8

Earlier quoted context omitted.

Seriously? Google is a private cache of the web. That is the problem.

Google doesn’t give anyone access to said cache. I mean one crawler with a shared api among competitors. So exactly the same as the public cache, but run my a private company and accessed for a small fee

> Google doesn’t give anyone access to said cache.

It would also be useful for deep searches, exceeding the 1000 result limit, empowering all sorts of NLP applications.

Post reply on HN