Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bypassing website anti-scraping protections

kb.apify.com

Post reply on HN
Print view View original

51–60 of 121 posts

Re: Bypassing website anti-scraping protections

#51

Post by dredmorbius »

Since people are asking "why would you do such a thing" or insinuating that scraping need only be to compete somehow with Google, I'll present a use I've found quite interesting, that doesn't seek to replicate or replace Google search, and which hasn't been readily attainable other than by scraping Google search results, in part. The tool I've used (crude, but reasonably effective) has applied numerous attempts to work around bot-detection, some modestly effective. (Rate-limiting most especially.)

I've found the practice of looking at search-term frequency, across a domain or set of domains (using the "site"" Google search filter) to be useful, for example the "Top 100 Global Thinkers" report linked below.

It uses 100 search terms -- "global thinkers" identified by Foreign Policy magazine -- searched across a set of about 100 domains and TLDs, largely social media, various journalism (newspaper / magazine), and a few institutional sites, as well as selected national and other top-level domains. The result is an interesting profile of where more robust online discussion or commentary might be found.

https://www.reddit.com/r/dredmorbius/comments/3hp41w/trackin...

The full report requires running roughly 100 x 100, or 10,000, Google searches. I'm finding that it's necessary to space these ~5-10 minutes apart, which means that the full analysis takes over a month of wall-clock time, from a single IP.

I've considered several possible follow-ups to this study, including more or alternate domains, different keywords, and various other variants, but both the run-time and codeing to bypass bot-detection put me off this.

I've tried reaching out to Googlers I know to see if there's any possible alternative means of acquiring this information, to no avail. I've also looked for various research interfaces or APIs, with no joy.

DuckDuckGo and other search sites don't have the rate-limiting (I've used them for other purposes), but also don't have the (granted, often very inaccurate / imprecise) match-counts which Google offers.

Putting this out there both as an example and a request for suggestions as to how I might improve or modify the process.

Re: Bypassing website anti-scraping protections

#52

Post by jocoda »

cm2187 wrote: post #24
dx034 wrote: post #10

Earlier quoted context omitted.

The rule set must be more complex. I often use VPN which results in captchas on many pages but I never get one on Google. I guess the 300 queries/IP only count if other parameters indicate crawling.

But it's a bit clunky. I was running searches through an embedded webbrowser in a c# application, which is really an embedded internet explorer and was very quickly presented with a captcha. It was a human viewing the results, but a script constructing the query string, but that was enough the be labelled as a crawler.

often all it takes is to be using webbrowser. I would love to use webbrowser for small search utilities because it's so easy to use, but it seems to be a magnet for problems.

Re: Bypassing website anti-scraping protections

#53

Post by danmg »

greglindahl wrote: post #44
userbinator wrote: post #3

For example, for google.com, you can typically make only around 300 requests per day, and if you reach this limit, you will see a CAPTCHA instead of search results. 300 is pretty easy to achieve if you're "Googling hard enough" (make 5 slightly different queries, go through the 20 pages of results it's willing to show you, repeat 3 times...), and I've seen it trigger far before that if you are searching for more obsc…

From my experience at blekko, 99.9% of the "people" who go deep into the results pages for a single query are actually bots. You're a very unusual user, and there are a lot of bots.

I run up against this all the time. My browser is fast from blocking all third party trackers and scripts. My searches are faster, so it thinks they're automated.

Re: Bypassing website anti-scraping protections

#54

Post by greglindahl »

jardah wrote: post #35
learc83 wrote: post #31

A company I consulted for was using a paid API to handle search. Despite the fact that the entire site was available in an easy to scrape XML format, scrapers kept using the search feature. They were trying very hard to overcome my countermeasures--they had a seemingly limitless pool of IPs, they were rotating user agent strings, and they tried to randomize search behavior. Everytime I implemented a new countermeasur…

That is kinda sad to hear. The approach should always be to go through the path of least resistance and smallest effect on the website. So for example, if a company has API that can be used instead of scraping their website, then it's always preferred to use the API. Same would go for the XML you mentioned. It's bad that not everyone works like this; there are quite a lot of people who would rather brute-force a solu…

The path of least resistance for the bots appears to be that they have a tool that scrapes search results, and nothing to talk to an API.

Re: Bypassing website anti-scraping protections

#55

Post by recursive »

throwawaymath wrote: post #13
icebraining wrote: post #8

Earlier quoted context omitted.

Google supports consensual scraping, and respects sites which opt-out (using robots.txt) just like they have. It's no more ironic than someone selling a product they don't happen to use themselves.

I think there's a credible argument that it's not purely consensual. Websites are forced to allow search engines with a lot of market share to scrape them or they won't be found. No matter how well-intentioned you are, if you write your own scraper and have it abide by robots.txt, you'll never get nearly as many resources as Google or Bing. Many websites approve only their scrapers and ban everything else outright. I…

I didn't consent to all this debt. It was just not really easy to say no to all these great credit cards.

Re: Bypassing website anti-scraping protections

#56

Post by AznHisoka »

0x00000000 wrote: post #23
userbinator wrote: post #3

For example, for google.com, you can typically make only around 300 requests per day, and if you reach this limit, you will see a CAPTCHA instead of search results. 300 is pretty easy to achieve if you're "Googling hard enough" (make 5 slightly different queries, go through the 20 pages of results it's willing to show you, repeat 3 times...), and I've seen it trigger far before that if you are searching for more obsc…

Instagram is the worst I have come across. If you are on a page with 1000+ pictures trying to find something near the bottom, you have to let it load each new group sequentially, then after a while it starts timing you out for like 60 seconds or longer every couple times you load more. God forbid you accidentally navigate away while scrolling you have to start all over again from the top. Due to recent events it seem…

Linkedin is even worse, imo. Go to any random company page and it'll show you a page, asking you to login. Refresh it again and it'll show you the page, without asking you to login.

Re: Bypassing website anti-scraping protections

#57

Post by fauigerzigerk »

vinceguidry wrote: post #5
cm2187 wrote: post #2

There is an irony in google preventing web scraping given that their business is pretty much built on web scraping.

Why is there irony in that? Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. Google protecting its site from scraping means you can't compete with Google using Google's own resources . That said, automated research fascinates me, I wouldn't want to scrape Google to make my own Google, but rather to make private repositories of information that I can then quer…

>Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google.

Unfortunately that is not the case. Many paywalled sites will let googlebot index their content but block other crawlers.

They may have good reasons for doing that in some cases, but as a consequence the level playing field you're talking about no longer exists.

Also, the purpose of using Google as part of some automated process is usually not to compete with Google's search engine, but to complete some specific and limited task.

I don't understand why Google does not have a general search API offering. I'm sure many people would happily pay for it.

Re: Bypassing website anti-scraping protections

#58

Post by ryanwaggoner »

learc83 wrote: post #31

A company I consulted for was using a paid API to handle search. Despite the fact that the entire site was available in an easy to scrape XML format, scrapers kept using the search feature. They were trying very hard to overcome my countermeasures--they had a seemingly limitless pool of IPs, they were rotating user agent strings, and they tried to randomize search behavior. Everytime I implemented a new countermeasur…

Maybe it was sabotage.

Re: Bypassing website anti-scraping protections

#59

Post by unreal37 »

learc83 wrote: post #31

A company I consulted for was using a paid API to handle search. Despite the fact that the entire site was available in an easy to scrape XML format, scrapers kept using the search feature. They were trying very hard to overcome my countermeasures--they had a seemingly limitless pool of IPs, they were rotating user agent strings, and they tried to randomize search behavior. Everytime I implemented a new countermeasur…

You had a paid API, and people wanted the information for free....

Not unexpected I guess.

Re: Bypassing website anti-scraping protections

#60

Post by unreal37 »

matthewmacleod wrote: post #22

On one hand, it does make a lot of sense that many web publishers want to keep people from scraping content, given the way that it's often used nefariously, to violate copyright, or for spam purposes. But there are totally legitimate reasons to scrape as well. Altmetric ( https://www.altmetric.com ), which is the company I work for, tracks links to scientific research. So when someone on e.g. Twitter links to a page…

The web is not becoming closed off from users. It's becoming hostile to bots. Not the same.
Post reply on HN