Live data from Hacker News

Bypassing website anti-scraping protections

kb.apify.com

31–40 of 121 posts

Re: Bypassing website anti-scraping protections

#31
A company I consulted for was using a paid API to handle search.

Despite the fact that the entire site was available in an easy to scrape XML format, scrapers kept using the search feature.

They were trying very hard to overcome my countermeasures--they had a seemingly limitless pool of IPs, they were rotating user agent strings, and they tried to randomize search behavior.

Everytime I implemented a new countermeasure they'd try to find a way around it. It was maddening because we made everything available for them through the XML feed. They just wouldn't use it.

Re: Bypassing website anti-scraping protections

#32

I find it odd how little a basic principle enters into this: don't do something to someone when they make it clear they don't want you to do it.

Eh, the principle might be good (though it's not odd that not everyone shares the same principles), but one can hold it and still have exceptions. For example, what about a governmental institution or a public company¹? What about a semi-public company, like a monopolist utility? What if the uploader of the data is OK with it, but the site hoster prevents it?

¹ in the sense of owned by the State, not listed on the stock market

Re: Bypassing website anti-scraping protections

#34
post #2

There is an irony in google preventing web scraping given that their business is pretty much built on web scraping.

Why is there irony in that? Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. Google protecting its site from scraping means you can't compete with Google using Google's own resources . That said, automated research fascinates me, I wouldn't want to scrape Google to make my own Google, but rather to make private repositories of information that I can then quer…

> Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google.

They cannot. Googlebot & some other search engine bots (like Bing's & Yandex's) get special treatment in various websites. This includes things like ban on non-whitelisted scrapers & bypassing paywalls. If you are not already established player in the field, you would not get able to scrape the same websites as the established players can.

Re: Bypassing website anti-scraping protections

#35
post #31

A company I consulted for was using a paid API to handle search. Despite the fact that the entire site was available in an easy to scrape XML format, scrapers kept using the search feature. They were trying very hard to overcome my countermeasures--they had a seemingly limitless pool of IPs, they were rotating user agent strings, and they tried to randomize search behavior. Everytime I implemented a new countermeasur…

That is kinda sad to hear. The approach should always be to go through the path of least resistance and smallest effect on the website. So for example, if a company has API that can be used instead of scraping their website, then it's always preferred to use the API. Same would go for the XML you mentioned.

It's bad that not everyone works like this; there are quite a lot of people who would rather brute-force a solution than think about it.

Re: Bypassing website anti-scraping protections

#36
post #4

For example, for google.com, you can typically make only around 300 requests per day, and if you reach this limit, you will see a CAPTCHA instead of search results. 300 is pretty easy to achieve if you're "Googling hard enough" (make 5 slightly different queries, go through the 20 pages of results it's willing to show you, repeat 3 times...), and I've seen it trigger far before that if you are searching for more obsc…

Yes it is, and since it's IP based, it's even easier if you are for example working from an office and there are multiple people using google. But that is why they only show recaptcha, you fill it in and you will get extemption cookie for 30 more requests :D

> But that is why they only show recaptcha, you fill it in and you will get extemption cookie for 30 more requests :D

Does that actually work? Whenever I have been searching some obscure things and managed to get the captcha after 6-10 pages, it just goes to loop where it keeps giving it constantly. Though it stops giving it if I change the search terms.

Re: Bypassing website anti-scraping protections

#37
post #14

Has there been precedent established whether bypassing anti-scraping does or does not violate the CFAA?

The generally-accepted precedent is that yes, unwanted scraping violates the CFAA. Given the age of the problem, the case law is still developing around it, but there have been many high-profile scraping cases, and the scraper almost always loses.

The reality is that the CFAA is extremely broad and if we want to protect "scraping", better termed something like "data preservation" or "data recovery", we need to change the CFAA, copyright, and the applicability of EULAs (which effectively work to plug any tiny leak that someone may've found through the CFAA-copyright combo).

Copyright itself makes it effectively illegal to read a web page without the owner's consent, even if a) there is no trespass/unauthorized access (CFAA); and b) there is no infringement in the actual content extracted. This occurs because the markup and other necessary supporting material around a page is a copyrighted work, and just reading it into memory and then immediately discarding is considered sufficiently tangible to infringe on the copyrighted work.

This is called the "RAM Copy Doctrine", and it has been [mis]applied to scrapers many times. In Facebook v. Power Ventures, it was used to stop a startup from helping Facebook users extract their own content. That founder was left owing $3M in damages to Facebook.

LinkedIn v. Hi5 is the most notable recent exception, but those rulings seem to be pure judicial activism unsupported by precedent or really any legal underpinnings, and will surely be overturned on appeal.

For every high-profile LinkedIn v. Hi5-style success, there are a good number of losses. It is fairly routine now after 3Taps.

IANAL, but my SaaS business, which depended on a key piece of scraped data, was destroyed by a legal threat from a Fortune 100.

Re: Bypassing website anti-scraping protections

#38

I find it odd how little a basic principle enters into this: don't do something to someone when they make it clear they don't want you to do it.

Eh, the principle might be good (though it's not odd that not everyone shares the same principles), but one can hold it and still have exceptions. For example, what about a governmental institution or a public company¹? What about a semi-public company, like a monopolist utility? What if the uploader of the data is OK with it, but the site hoster prevents it? ¹ in the sense of owned by the State, not listed on the st…

I'm not saying there aren't legitimate reasons for writing scrapers. I've written plenty myself. It was just odd to see this disregarded entirely.

As for the commonality of principles, game theory explains most of them, so it isn't more surprising than that we all work with the same prime numbers, say. A simple principle of reciprocity will produce something along the lines of "respect other people's wishes".

Re: Bypassing website anti-scraping protections

#39
post #34

Earlier quoted context omitted.

Why is there irony in that? Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. Google protecting its site from scraping means you can't compete with Google using Google's own resources . That said, automated research fascinates me, I wouldn't want to scrape Google to make my own Google, but rather to make private repositories of information that I can then quer…

> Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. They cannot. Googlebot & some other search engine bots (like Bing's & Yandex's) get special treatment in various websites. This includes things like ban on non-whitelisted scrapers & bypassing paywalls. If you are not already established player in the field, you would not get able to scrape the same websites…

As I understand it, this was the rationale behind the courts' decision to prohibit LinkedIn from banning people from scraping public profiles.

Basically it was anti competitive to grant certain privileges to major players around 'public data,' but to block smaller players.

No telling if/when ramifications from that decision (last year) hit existing anti scraping measures, though.

Re: Bypassing website anti-scraping protections

#40
> there are already anti-scraping solutions on the market that can detect its usage based on a variable it puts into the browser's window.navigator property. Thankfully, we have developed a solution which removes the property from the web browser

Does anyone know what exactly the property in question is?

Post reply on HN