I find it odd how little a basic principle enters into this: don't do something to someone when they make it clear they don't want you to do it.
And I get downvoted as a norm breaker for mentioning norm breaking. Hacker News becomes Fight Club.
Bypassing website anti-scraping protections
21–30 of 121 posts
Re: Bypassing website anti-scraping protections
#22On one hand, it does make a lot of sense that many web publishers want to keep people from scraping content, given the way that it's often used nefariously, to violate copyright, or for spam purposes.
But there are totally legitimate reasons to scrape as well. Altmetric (https://www.altmetric.com), which is the company I work for, tracks links to scientific research. So when someone on e.g. Twitter links to a page on nature.com, we want to scrap the page they linked to and figure out which paper they are talking about (if any). Academic publishers can be particularly sensitive to scraping, making the endeavour much more work than it needs to be.
It's a real shame that the web has moved to be so closed off in many ways.
Re: Bypassing website anti-scraping protections
#23For example, for google.com, you can typically make only around 300 requests per day, and if you reach this limit, you will see a CAPTCHA instead of search results. 300 is pretty easy to achieve if you're "Googling hard enough" (make 5 slightly different queries, go through the 20 pages of results it's willing to show you, repeat 3 times...), and I've seen it trigger far before that if you are searching for more obsc…
Instagram is the worst I have come across. If you are on a page with 1000+ pictures trying to find something near the bottom, you have to let it load each new group sequentially, then after a while it starts timing you out for like 60 seconds or longer every couple times you load more. God forbid you accidentally navigate away while scrolling you have to start all over again from the top.
Due to recent events it seems they got scared, locked down their API, then tightened down the request limit to prevent scraping to the point it is hardly usable on desktop anyway.
Re: Bypassing website anti-scraping protections
#24For example, for google.com, you can typically make only around 300 requests per day, and if you reach this limit, you will see a CAPTCHA instead of search results. 300 is pretty easy to achieve if you're "Googling hard enough" (make 5 slightly different queries, go through the 20 pages of results it's willing to show you, repeat 3 times...), and I've seen it trigger far before that if you are searching for more obsc…
The rule set must be more complex. I often use VPN which results in captchas on many pages but I never get one on Google. I guess the 300 queries/IP only count if other parameters indicate crawling.
But it's a bit clunky. I was running searches through an embedded webbrowser in a c# application, which is really an embedded internet explorer and was very quickly presented with a captcha. It was a human viewing the results, but a script constructing the query string, but that was enough the be labelled as a crawler.
Re: Bypassing website anti-scraping protections
#25Earlier quoted context omitted.
Why is there irony in that? Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. Google protecting its site from scraping means you can't compete with Google using Google's own resources . That said, automated research fascinates me, I wouldn't want to scrape Google to make my own Google, but rather to make private repositories of information that I can then quer…
> Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. I don't think that making a scraper will make you competitive with Google. If you can make a site ranking algorithm that compete's with google, on the other hand, you might have a chance
The site ranking algorithm is a solved problem.
The one reason Google is competitive is due to them taking advantage of the cheap labour that keeps track of ranking manipulation.
Luckily most of the search problems have nothing to do with ranking manipulation.
Re: Bypassing website anti-scraping protections
#26Earlier quoted context omitted.
That’s precisely why they don’t want you cheating the hard part and just storing the results. It makes sense to me. Work on your own machine learning if you want good results.
You could argue that Google should work on their own knowledge database instead of learning from other people's content and/or presenting other people's content in their own frontends (shopping etc)...
This is what Common Crawl does: http://commoncrawl.org/. I think more people should know about it.
Re: Bypassing website anti-scraping protections
#27I find it odd how little a basic principle enters into this: don't do something to someone when they make it clear they don't want you to do it.
Move fast and break things, like norms
Re: Bypassing website anti-scraping protections
#28There is an irony in google preventing web scraping given that their business is pretty much built on web scraping.
Why is there irony in that? Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. Google protecting its site from scraping means you can't compete with Google using Google's own resources . That said, automated research fascinates me, I wouldn't want to scrape Google to make my own Google, but rather to make private repositories of information that I can then quer…
> to make private repositories of information that I can then query efficiently
You and me both :)
I still haven't gotten around to do much about it, but for example one thing I've been thinking about is to have my system integrated with my desktop so that it has some situational context.
For example, it would look at the programs that I have currently running.
Let's say that it saw that I had PyCharm open where I was editing some Python 3 files. Furthermore I also had Vim open where I was editing some HTML, CSS and JavaScript files.
It would maintain a list of all items that had been in focus during the previous 30 minutes or something.
When I then searched for let's say sort list it would look at the list and see that most recently I had been editing a Python file in PyCharm so result number 1 would be how to sort a list in Python 3. Before that I had also focused Vim with a JS file, so sorting arrays in JS would be result number 2.
Results:
1. Python 3. Sort list "a_list". In-place: a_list.sort(). Build new sorted list from iterable: b_list = sorted(a_list).
2. JavaScript. Sort array "an_array". In-place: a_list.sort(). Create a new shallow copy and sort array: let another_array = an_array.concat().sort().
And if the system was even smarter, it would also be able to know details about what I'd been doing. For example it could see that while editing a JavaScript file I had most recently been writing code that was doing some operations with WebGL, and before that I was editing code that was changing style properties and before that something that was working with Canvas, so if I then search for blend, it would use this information.
Results:
1. WebGL Lesson 8 – the depth buffer, transparency and blending. http://learningwebgl.com/blog/?p=859
2. Basics of CSS Blend Modes. https://css-tricks.com/basics-css-blend-modes/
3. CanvasRenderingContext2D.globalCompositeOperation. https://developer.mozilla.org/en-US/docs/Web/API/CanvasRende...
Something like that.
And because it's for the limited amount of things that I am interested in and developed for myself only (as opposed to trying to give super relevant information for every person in the world), it might be doable to some extent.
Here is a book that might be of interest to you; Relevant Search. https://www.manning.com/books/relevant-search. I bought a copy myself but have yet to read it.
Re: Bypassing website anti-scraping protections
#29There is an irony in google preventing web scraping given that their business is pretty much built on web scraping.
Shameless plug: it’s why we made SerpApi! (https://serpapi.com)
Re: Bypassing website anti-scraping protections
#30Earlier quoted context omitted.
That’s precisely why they don’t want you cheating the hard part and just storing the results. It makes sense to me. Work on your own machine learning if you want good results.
There is no such thing as cheating, only staying within boundaries that don't land you in jail or sued in your own jurisdiction. If you can get an edge by using Google's own data, do so.
Bing bing bing!
Er, ughm. I mean,
Ding ding ding!