What sorts of real-world and legitimate/ethical use cases are there for wholesale repeated scraping?
Internet archival initiatives are a legitimate use case, as well as things like sneakernet in Cuba.
How to scrape anything on the web and not get caught
31–40 of 57 posts
Re: How to scrape anything on the web and not get caught
#32What sorts of real-world and legitimate/ethical use cases are there for wholesale repeated scraping?
Scraping websites like AliExpress and Ebay for trademark and brand infringement. There are many vendors who sell fake tat on the internet, and there are companies out there who track it down on behalf of the brand owners.
These companies tend to have armies of lawyers who can swat away even the likes of eBay when it comes to justifying web scraping. Nevertheless, the work required is still the same tedium that others deal with: CAPTCHAs; throttling; IP bans; etc.
Re: How to scrape anything on the web and not get caught
#33What sorts of real-world and legitimate/ethical use cases are there for wholesale repeated scraping?
I'm kind of interested in an answer to this as well.
I know the typical "travel site" or "comparison shop" use case.
There is also the "darn it, I want this" use case.
However, automated, periodic web scraping that mutates (I.e. ticket or reservation grabbing bots) has always felt a bit squicky to me in the same way DNS squatting does.
Re: How to scrape anything on the web and not get caught
#34That's a very low-quality article, in my opinion. It takes an entire article to show how to use a simple tool and how to fetch a list of proxies, uses Makefile when a shell script would do just fine, and exaggerates the title.
Re: How to scrape anything on the web and not get caught
#35What sorts of real-world and legitimate/ethical use cases are there for wholesale repeated scraping?
Statistics I'd say is one of the useful cases of scraping.
Back in 2013, a guy scraped the results of about 150,000 students giving their 10th grade finals of a particular examination board in India. He showed that not only was there no privacy of student's marks because the roll numbers were all linearly incremented, but there was also mass-scale manipulation of marks going on.
The concept is simple but it's a very interesting read.
https://deedy.quora.com/Hacking-into-the-Indian-Education-Sy...
I was one of the 150,000 kids that gave those exams back in 2013 :)
Re: How to scrape anything on the web and not get caught
#36What sorts of real-world and legitimate/ethical use cases are there for wholesale repeated scraping?
Scrape government data... it’s all free but try to find it in any downloadable and consistent format.
Re: How to scrape anything on the web and not get caught
#37What sorts of real-world and legitimate/ethical use cases are there for wholesale repeated scraping?
Statistics I'd say is one of the useful cases of scraping. Back in 2013, a guy scraped the results of about 150,000 students giving their 10th grade finals of a particular examination board in India. He showed that not only was there no privacy of student's marks because the roll numbers were all linearly incremented, but there was also mass-scale manipulation of marks going on. The concept is simple but it's a very…
Facinating read, I cannot find out more about what happened to him after he was accused of 'hacking' the govt systems, do you have any more sources that sheds some light on that?
Re: How to scrape anything on the web and not get caught
#38I use paid service proxy bonanza ($12/mo for 2 IPs), and build my own as well using squid ($5/mo DigitalOcean).
You pay more per IP than per server!? Why don't you get 3 DO instances then?
I can't speak for that particular service, but in other similar ones I've looked at and used in the past, you're paying for a certain number of IPs _at a given time_.
So, for instance, they have a pool of servers that have 1000 IPs available. Your account allows connections to go out over 2 of those at a time. If something happens (like one gets banned by whatever service you're scraping), you can get a different set of 2 IPs and keep moving.
While you're still paying a relatively high price for what you're consuming (predominantly bandwidth in this case), you're paying for the flexibility.
Re: How to scrape anything on the web and not get caught
#39This article has the lowest quality ever seen on HN. The title is hugely exaggerated.
Using a list of proxies and hope that is enough to scrape _anything_ on web?
Re: How to scrape anything on the web and not get caught
#40That's a very low-quality article, in my opinion. It takes an entire article to show how to use a simple tool and how to fetch a list of proxies, uses Makefile when a shell script would do just fine, and exaggerates the title.
I came here to make this same comment. This article barely begins to scratch the surface on this topic. Effective, high volume scraping involves more than just rotating through a list of proxies.