Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bypassing website anti-scraping protections

kb.apify.com

Post reply on HN
Print view View original

11–20 of 121 posts

Re: Bypassing website anti-scraping protections

#11

Post by jamra »

jedimastert wrote: post #7
vinceguidry wrote: post #5

Earlier quoted context omitted.

Why is there irony in that? Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. Google protecting its site from scraping means you can't compete with Google using Google's own resources . That said, automated research fascinates me, I wouldn't want to scrape Google to make my own Google, but rather to make private repositories of information that I can then quer…

> Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. I don't think that making a scraper will make you competitive with Google. If you can make a site ranking algorithm that compete's with google, on the other hand, you might have a chance

That’s precisely why they don’t want you cheating the hard part and just storing the results. It makes sense to me. Work on your own machine learning if you want good results.

Re: Bypassing website anti-scraping protections

#12

Post by jardah »

dx034 wrote: post #10
userbinator wrote: post #3

For example, for google.com, you can typically make only around 300 requests per day, and if you reach this limit, you will see a CAPTCHA instead of search results. 300 is pretty easy to achieve if you're "Googling hard enough" (make 5 slightly different queries, go through the 20 pages of results it's willing to show you, repeat 3 times...), and I've seen it trigger far before that if you are searching for more obsc…

The rule set must be more complex. I often use VPN which results in captchas on many pages but I never get one on Google. I guess the 300 queries/IP only count if other parameters indicate crawling.

Yea, it was a very general example, since there is at least one rule that is based on rate limiting too, and this 300/IP limit is what have seen on average.

Re: Bypassing website anti-scraping protections

#13

Post by throwawaymath »

icebraining wrote: post #8
cm2187 wrote: post #6

Earlier quoted context omitted.

The irony is the "do as I say, not as I do".

Google supports consensual scraping, and respects sites which opt-out (using robots.txt) just like they have. It's no more ironic than someone selling a product they don't happen to use themselves.

I think there's a credible argument that it's not purely consensual. Websites are forced to allow search engines with a lot of market share to scrape them or they won't be found.

No matter how well-intentioned you are, if you write your own scraper and have it abide by robots.txt, you'll never get nearly as many resources as Google or Bing. Many websites approve only their scrapers and ban everything else outright.

I don't have anything against the large search engines, it's just not really easy to say no to their scrapers for most websites.

Re: Bypassing website anti-scraping protections

#15

Post by nerdponx »

userbinator wrote: post #3

For example, for google.com, you can typically make only around 300 requests per day, and if you reach this limit, you will see a CAPTCHA instead of search results. 300 is pretty easy to achieve if you're "Googling hard enough" (make 5 slightly different queries, go through the 20 pages of results it's willing to show you, repeat 3 times...), and I've seen it trigger far before that if you are searching for more obsc…

Use DDG. It's fine for most things. Use Google as a fallback.

Re: Bypassing website anti-scraping protections

#16

Post by ksahin »

Asooka wrote: post #14

Has there been precedent established whether bypassing anti-scraping does or does not violate the CFAA?

It's a complex subject.

For example, the Linkedin case : https://arstechnica.com/tech-policy/2017/08/court-rejects-li...

Craiglist sued some companies too.

To my understanding, scraping can be legal if it's done properly, meaning not sending too many requests at the same time, and if it does not affect the underlying infrastructure.

It seems like in the US or in Europe, even if there is any anti-bot / anti-scraping section in the website's TOS, public data can be scraped. Sometimes, even "private" data can be extracted using bots. For example, lots of "bank account aggregators" has won lawsuits against banks.

Re: Bypassing website anti-scraping protections

#17

Post by epanchin »

userbinator wrote: post #3

For example, for google.com, you can typically make only around 300 requests per day, and if you reach this limit, you will see a CAPTCHA instead of search results. 300 is pretty easy to achieve if you're "Googling hard enough" (make 5 slightly different queries, go through the 20 pages of results it's willing to show you, repeat 3 times...), and I've seen it trigger far before that if you are searching for more obsc…

With a search as you type feature (Google instant?) that limit could be hit in a few searches...

Re: Bypassing website anti-scraping protections

#18

Post by anc84 »

jamra wrote: post #11
jedimastert wrote: post #7

Earlier quoted context omitted.

> Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. I don't think that making a scraper will make you competitive with Google. If you can make a site ranking algorithm that compete's with google, on the other hand, you might have a chance

That’s precisely why they don’t want you cheating the hard part and just storing the results. It makes sense to me. Work on your own machine learning if you want good results.

You could argue that Google should work on their own knowledge database instead of learning from other people's content and/or presenting other people's content in their own frontends (shopping etc)...

Re: Bypassing website anti-scraping protections

#20

Post by toomuchtodo »

jamra wrote: post #11
jedimastert wrote: post #7

Earlier quoted context omitted.

> Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. I don't think that making a scraper will make you competitive with Google. If you can make a site ranking algorithm that compete's with google, on the other hand, you might have a chance

That’s precisely why they don’t want you cheating the hard part and just storing the results. It makes sense to me. Work on your own machine learning if you want good results.

There is no such thing as cheating, only staying within boundaries that don't land you in jail or sued in your own jurisdiction. If you can get an edge by using Google's own data, do so.
Post reply on HN