Bypassing website anti-scraping protections
kb.apify.com
Bypassing website anti-scraping protections
1–10 of 121 posts
Re: Bypassing website anti-scraping protections
#2There is an irony in google preventing web scraping given that their business is pretty much built on web scraping.
Re: Bypassing website anti-scraping protections
#3For example, for google.com, you can typically make only around 300 requests per day, and if you reach this limit, you will see a CAPTCHA instead of search results.
300 is pretty easy to achieve if you're "Googling hard enough" (make 5 slightly different queries, go through the 20 pages of results it's willing to show you, repeat 3 times...), and I've seen it trigger far before that if you are searching for more obscure things. It seems almost hostile to those searching for IC part numbers, specific and very exact phrases, and just "non mainstream" content in general.
How sad it is then, that we are told and have internalised the notion that we should use search engines like Google to find things, and yet it prevents us from "trying too hard" to find what we're looking for...
Re: Bypassing website anti-scraping protections
#4For example, for google.com, you can typically make only around 300 requests per day, and if you reach this limit, you will see a CAPTCHA instead of search results. 300 is pretty easy to achieve if you're "Googling hard enough" (make 5 slightly different queries, go through the 20 pages of results it's willing to show you, repeat 3 times...), and I've seen it trigger far before that if you are searching for more obsc…
Yes it is, and since it's IP based, it's even easier if you are for example working from an office and there are multiple people using google.
But that is why they only show recaptcha, you fill it in and you will get extemption cookie for 30 more requests :D
Re: Bypassing website anti-scraping protections
#5There is an irony in google preventing web scraping given that their business is pretty much built on web scraping.
Why is there irony in that? Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. Google protecting its site from scraping means you can't compete with Google using Google's own resources.
That said, automated research fascinates me, I wouldn't want to scrape Google to make my own Google, but rather to make private repositories of information that I can then query efficiently. I would love to find any kind of scriptable search engine access, paid or free. Not entirely sure how to look though.
Re: Bypassing website anti-scraping protections
#6There is an irony in google preventing web scraping given that their business is pretty much built on web scraping.
Why is there irony in that? Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. Google protecting its site from scraping means you can't compete with Google using Google's own resources . That said, automated research fascinates me, I wouldn't want to scrape Google to make my own Google, but rather to make private repositories of information that I can then quer…
The irony is the "do as I say, not as I do".
Re: Bypassing website anti-scraping protections
#7There is an irony in google preventing web scraping given that their business is pretty much built on web scraping.
Why is there irony in that? Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. Google protecting its site from scraping means you can't compete with Google using Google's own resources . That said, automated research fascinates me, I wouldn't want to scrape Google to make my own Google, but rather to make private repositories of information that I can then quer…
> Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google.
I don't think that making a scraper will make you competitive with Google. If you can make a site ranking algorithm that compete's with google, on the other hand, you might have a chance
Re: Bypassing website anti-scraping protections
#8Earlier quoted context omitted.
Why is there irony in that? Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. Google protecting its site from scraping means you can't compete with Google using Google's own resources . That said, automated research fascinates me, I wouldn't want to scrape Google to make my own Google, but rather to make private repositories of information that I can then quer…
The irony is the "do as I say, not as I do".
Google supports consensual scraping, and respects sites which opt-out (using robots.txt) just like they have. It's no more ironic than someone selling a product they don't happen to use themselves.
Re: Bypassing website anti-scraping protections
#9Earlier quoted context omitted.
Why is there irony in that? Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. Google protecting its site from scraping means you can't compete with Google using Google's own resources . That said, automated research fascinates me, I wouldn't want to scrape Google to make my own Google, but rather to make private repositories of information that I can then quer…
The irony is the "do as I say, not as I do".
Google's web scrapers obey robots.txt, you can stop Google from crawling your website if you want. Google doesn't want you crawling their website.
That word, I don't think it means what you think it means.
Re: Bypassing website anti-scraping protections
#10For example, for google.com, you can typically make only around 300 requests per day, and if you reach this limit, you will see a CAPTCHA instead of search results. 300 is pretty easy to achieve if you're "Googling hard enough" (make 5 slightly different queries, go through the 20 pages of results it's willing to show you, repeat 3 times...), and I've seen it trigger far before that if you are searching for more obsc…
The rule set must be more complex. I often use VPN which results in captchas on many pages but I never get one on Google. I guess the 300 queries/IP only count if other parameters indicate crawling.