Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Only Google is really allowed to crawl the web

knuckleheads.club

Post reply on HN
Print view View original

251–260 of 365 posts

Re: Only Google is really allowed to crawl the web

#251

Post by quantumofalpha »

p-sharma wrote: post #154
quantumofalpha wrote: post #103

Earlier quoted context omitted.

Ignoring robots.txt is trivial, that's why some(many?) sites enforce it by verifying source IP and recognize Googlebot from its IP addresses - how will you get access to one of those?

What does "recognize Googlebot from its IP addresses" mean? If I'm a human and I access a site, I have some other IP than Googlebot, how should this side know if I'm a human or knuckleheadsbot?

if you're claiming to be User-Agent: Googlebot, but your IP doesn't seem like it belongs to Google, don't you think it's a clear sign that you're FAKING IT?

The check itself could be implemented for example with ASN or reverse DNS lookup or hard-coding known Google's IP ranges (though that's prone to become stale)

Re: Only Google is really allowed to crawl the web

#252

Post by darkwater »

treis wrote: post #50
tyingq wrote: post #12

The bigger problem, to me, is not around crawling. It's the asymmetrical power Google has after crawling. Google is obviously on a mission to keep people on Google owned properties. So, they take what they crawl and find a way to present that to the end user without anyone needing to visit the place that data came from. Airlines are a good example. If you search for flight status for a particular flight, Google prese…

>However, that sort of search used to (most times) lead to a visit to the airline web site. I don't think that's correct. In the old days you'd either call a travel agent or use an aggregator like expedia. Google muscles out intermediaries like Expedia, Yelp, and so on. It's not likely much better or worse for the end user or supplier. Just swapping one middleman for another.

It's actually pretty different because another middleman can basically arise only if it's a big success in the iOS App Store because coming up in Google searches would be impossible and more or less the same in the Play Store. So, Google is not just yet another intermediary.

Re: Only Google is really allowed to crawl the web

#253

Post by wbl »

tyingq wrote: post #12

The bigger problem, to me, is not around crawling. It's the asymmetrical power Google has after crawling. Google is obviously on a mission to keep people on Google owned properties. So, they take what they crawl and find a way to present that to the end user without anyone needing to visit the place that data came from. Airlines are a good example. If you search for flight status for a particular flight, Google prese…

Does the concergie of a hotel take anything away when he informs you that your flight has been delayed?

Re: Only Google is really allowed to crawl the web

#254

Post by f430 »

smarx007 wrote: post #249
TameAntelope wrote: post #230

Earlier quoted context omitted.

How hard is it to ask Cloudflare to let you crawl?

It's not Cloudflare who is deciding it. It's the website owners who request things like "Super Bot Fight Mode". I never enable such things on my CF properties. Mostly it's people who manage websites with "valuable" content, e.g. shops with prices who desperately want to stop scraping by competitors.

I can say this will give a lot of businesses false sense of security. It is already bypassable.

the Web scraping technology that I am aware of has reached end game already: Unless you are prepared to authenticate every user/visitor to your website with a dollar sign, lobby congress to pass a bill to outlaw web scraping, you will not be able to stop web scraping in 2021 and beyond.

Re: Only Google is really allowed to crawl the web

#255

Post by foobar33333 »

jonas21 wrote: post #166
JackFr wrote: post #93

Earlier quoted context omitted.

So from the website's point of view there is no difference between 'crawling' and 'scraping'. Census.gov I assume has a ton of very useful information which is in the public domain which a host of potential companies could monetize by regularly scraping census.gov. Census.gov's purpose to make this information available to people is served by google, yahoo and bing. On the other hand if I have a business which is bas…

The census data is available for bulk download, mostly as CSV (for example [1]). Scraping census.gov is worse for both the Census Bureau (which might have to do an expensive database query for each page) and for the scraper (who has to parse the page). Blocking scrapers in robots.txt is more of a way of saying, "hey, you're doing it wrong." It's also worth noting that the original article is out of date. The current…

Scrapers don't care about robots.txt. I have scraped multiple websites in a previous job and the robots.txt means nothing. Bigger sites might detect and block you but most don't.

Re: Only Google is really allowed to crawl the web

#256

Post by 838812052807016 »

tyingq wrote: post #12

The bigger problem, to me, is not around crawling. It's the asymmetrical power Google has after crawling. Google is obviously on a mission to keep people on Google owned properties. So, they take what they crawl and find a way to present that to the end user without anyone needing to visit the place that data came from. Airlines are a good example. If you search for flight status for a particular flight, Google prese…

Standardized interoperability enables overall progress.

Every airline doesn't need their own webpage. They could all provide a standard API.

Re: Only Google is really allowed to crawl the web

#257

Post by ilaksh »

tyingq wrote: post #12

The bigger problem, to me, is not around crawling. It's the asymmetrical power Google has after crawling. Google is obviously on a mission to keep people on Google owned properties. So, they take what they crawl and find a way to present that to the end user without anyone needing to visit the place that data came from. Airlines are a good example. If you search for flight status for a particular flight, Google prese…

The way that the web has been fundamentally broken by Google and other companies is one of the reasons I am excited about an alternative protocol called Gemini. It doesn't replace the web entirely, but for basic things like exchanging information, it's great. https://gemini.circumlunar.space/

Re: Only Google is really allowed to crawl the web

#258

Post by bombcar »

cyberpunk wrote: post #248
gtm1260 wrote: post #233

Earlier quoted context omitted.

I'm not sure I agree with this. I think airline websites are so garbage filled that they've driven people to use the simple alternative of the google flights checkout. It's a bit of a vicious cycle, but In general most websites are so chock filled with crap that not having to click into them for real is a relief!

BA had some tracking request inline on the “payment processing” page which when blocked by my pihole prevents me from ever getting to the confirmation page, just have to refresh your email and wait for the best. I have no idea how these companies, which make quite a decent amount of money at least up until 2020, can have such utterly poor sites. I once counted some 20+ redirects on a single request during this proces…

I don’t know what they’re doing but most every single sign on tool I’ve seen redirects 10-20 times during the sign on process (and then dumps you to the homepage to navigate your way back).

Re: Only Google is really allowed to crawl the web

#259

Post by johncolanduoni »

quitethelogic wrote: post #239
soheil wrote: post #17

Earlier quoted context omitted.

It's hilarious to think there exists people who think googlebot does not get special treatment from website operators. Here is an experiment you can do in a jiffy, write a script that crawls any major website and see how many URL fetches it takes before your IP gets blocked. Googlebot has a range of IP addresses that it publicly announces so websites can whitelist them.

> Googlebot has a range of IP addresses that it publicly announces so websites can whitelist them. Google says[1] they do not do this: "Google doesn't post a public list of IP addresses for website owners to allowlist." [1] https://developers.google.com/search/docs/advanced/crawling/...

From that same page they recommend using a reverse DNS lookup (and then a forward DNS lookup on the returned domain) to validate that it is google bot. So the effect is the same for anyone trying to impersonate googlebot (unless they can attack the DNS resolution of the site they’re scraping I guess).

Re: Only Google is really allowed to crawl the web

#260

Post by DocTomoe »

the_dege wrote: post #110
TheAdamAndChe wrote: post #51

Earlier quoted context omitted.

Are there any actual repercussions for just ignoring robots.txt?

Sometimes website admins will also try to report your ips to the service provider as a source of attacks (even if not true).

Given how often I've had misbehaving crawlers slow own servers in the early 2000s, I do not see how a crawler that disobeys robots.txt is not an attempted attack.
Post reply on HN