71–80 of 121 posts
For example, for google.com, you can typically make only around 300 requests per day, and if you reach this limit, you will see a CAPTCHA instead of search results. 300 is pretty easy to achieve if you're "Googling hard enough" (make 5 slightly different queries, go through the 20 pages of results it's willing to show you, repeat 3 times...), and I've seen it trigger far before that if you are searching for more obsc…
There is an irony in google preventing web scraping given that their business is pretty much built on web scraping.
Earlier quoted context omitted.
Why is there irony in that? Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. Google protecting its site from scraping means you can't compete with Google using Google's own resources . That said, automated research fascinates me, I wouldn't want to scrape Google to make my own Google, but rather to make private repositories of information that I can then quer…
>Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. Unfortunately that is not the case. Many paywalled sites will let googlebot index their content but block other crawlers. They may have good reasons for doing that in some cases, but as a consequence the level playing field you're talking about no longer exists. Also, the purpose of using Google as part of som…
Doesn't that infringe upon Google's own rules? I always thought Google didn't like it when sites served its crawler content that's different from what users get when they follow Google's link.
Has there been precedent established whether bypassing anti-scraping does or does not violate the CFAA?
It's a complex subject. For example, the Linkedin case : https://arstechnica.com/tech-policy/2017/08/court-rejects-li... Craiglist sued some companies too. To my understanding, scraping can be legal if it's done properly, meaning not sending too many requests at the same time, and if it does not affect the underlying infrastructure. It seems like in the US or in Europe, even if there is any anti-bot / anti-scraping s…
At this point you should just consider your HTML/HTTP interface an API because when you use headless browser technology readily available with any programming language it becomes exactly that.
Writing a site-specific browser has always been a fun project for me. It just pulls the information I want directly from my favorite websites. Maximum signal-to-noise ratio and I get ad blocking for free.
People think Javascript-based sites are safer, but it's in fact even easier to access the content because there's usually a programmatic interface available.
Earlier quoted context omitted.
>Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. Unfortunately that is not the case. Many paywalled sites will let googlebot index their content but block other crawlers. They may have good reasons for doing that in some cases, but as a consequence the level playing field you're talking about no longer exists. Also, the purpose of using Google as part of som…
>Many paywalled sites will let googlebot index their content but block other crawlers. Doesn't that infringe upon Google's own rules? I always thought Google didn't like it when sites served its crawler content that's different from what users get when they follow Google's link.
But it no longer works at all sites. Maybe the rule has been dropped now that paywalls are becoming more popular (with publishers that is)
Earlier quoted context omitted.
Yes it is, and since it's IP based, it's even easier if you are for example working from an office and there are multiple people using google. But that is why they only show recaptcha, you fill it in and you will get extemption cookie for 30 more requests :D
> But that is why they only show recaptcha, you fill it in and you will get extemption cookie for 30 more requests :D Does that actually work? Whenever I have been searching some obscure things and managed to get the captcha after 6-10 pages, it just goes to loop where it keeps giving it constantly. Though it stops giving it if I change the search terms.
Earlier quoted context omitted.
It's a complex subject. For example, the Linkedin case : https://arstechnica.com/tech-policy/2017/08/court-rejects-li... Craiglist sued some companies too. To my understanding, scraping can be legal if it's done properly, meaning not sending too many requests at the same time, and if it does not affect the underlying infrastructure. It seems like in the US or in Europe, even if there is any anti-bot / anti-scraping s…
The issue is that if you allowed all web scrapping you could DDOS websites and get a free out of jail card by telling "oh, we were simply scrapping some data and it glitched out".
For example, for google.com, you can typically make only around 300 requests per day, and if you reach this limit, you will see a CAPTCHA instead of search results. 300 is pretty easy to achieve if you're "Googling hard enough" (make 5 slightly different queries, go through the 20 pages of results it's willing to show you, repeat 3 times...), and I've seen it trigger far before that if you are searching for more obsc…
From my experience at blekko, 99.9% of the "people" who go deep into the results pages for a single query are actually bots. You're a very unusual user, and there are a lot of bots.