How Google’s Web Crawler Bypasses Paywalls • Hacker News Board

Re: How Google’s Web Crawler Bypasses Paywalls

#241

Post by trasc » Sun, Feb 21, 2016, 6:16 PM UTC

queeerkopf wrote: post #239

NKCSS wrote: post #217

Earlier quoted context omitted.

Google also specifies the ip ranges of their boys; just UA checking is sloppy

Hmmm, the following seems to contradict that; instead google recommends verification by DNS lookup: https://support.google.com/webmasters/answer/80553 'Google doesn't post a public list of IP addresses for webmasters to whitelist. This is because these IP address ranges can change, causing problems for any webmasters who have hard-coded them, so you must run a DNS lookup as described next.'

If anyone can tell me how to embed this particular DNS query into a firewall appliance ACL, please let me know. I have customers with Watchguard and Cisco gear who have intermittent trouble with Google's mail relay (formerly Postini). The solution is always to perform that record lookup and add the new IP address blocs.

Re: How Google’s Web Crawler Bypasses Paywalls

#242

Post by greglindahl » Sun, Feb 21, 2016, 6:46 PM UTC

dogma1138 wrote: post #38

greglindahl wrote: post #34

Earlier quoted context omitted.

I don't think that any server that random people can start a proxy on is in Google's SPF record.

not on the spf list, but on google's domain/ip blocks list.

The link you yourself posted says "The most effective means of finding the current range of Google IP addresses is to query Google's SPF record." While this is intended to be mailservers only, it has always appeared to me that the SPF list is all of their servers that 3rd parties can't run proxies on.

Re: How Google’s Web Crawler Bypasses Paywalls

#243

Post by mildweed » Mon, Feb 22, 2016, 6:03 AM UTC

Solution:

Content providers register a (yet-to-be-written) Google News API account, get an API key, with which Google indexes the site and the site recognizes as legit.