Could this be a solution?: run a website, and let your visitors do the crawling.
Won’t work. Adversaries can taint your data by sending back fake results.
Could multiple downloads and a "consensus" algorithm solve this problem?
How to scrape anything on the web and not get caught
21–30 of 57 posts
Re: How to scrape anything on the web and not get caught
#22I use paid service proxy bonanza ($12/mo for 2 IPs), and build my own as well using squid ($5/mo DigitalOcean).
Re: How to scrape anything on the web and not get caught
#23I use paid service proxy bonanza ($12/mo for 2 IPs), and build my own as well using squid ($5/mo DigitalOcean).
You pay more per IP than per server!? Why don't you get 3 DO instances then?
Re: How to scrape anything on the web and not get caught
#24I have no idea what tools are available for denying access to web scrapers, this I should know given that I have built a few websites and know what to do to get pages serving quickly. Somehow I missed the memo on how to set your site up to not be scraped. Is there an nginx setting for that? This could be interesting for people that do scrape sites to know too, what basic reasonable measures can one take beyond lookin…
Captchas are one (drastic) option.
Re: How to scrape anything on the web and not get caught
#25Earlier quoted context omitted.
Won’t work. Adversaries can taint your data by sending back fake results.
Could multiple downloads and a "consensus" algorithm solve this problem?
Kinda, but it’s far from trivial as you would need some sort of tolerance when comparing sites with dynamic content.
Re: How to scrape anything on the web and not get caught
#26Re: How to scrape anything on the web and not get caught
#27I had to scrape past an access limit per IP once, and just went with IPv6 addresses, of which IPv6 users have plenty. (This was almost five years ago, so it's conceivable that some services would have wisened up a bit and block the entire IPv6 block now.)
Re: How to scrape anything on the web and not get caught
#28I assume simple BFS and DFS traversal behavior shows up brightly in access logs, making detection more likely. Does it help to use Random First Search[1]? Or is it better to attempt emulating human actors (requires much more develop effort)?
Re: How to scrape anything on the web and not get caught
#29I have no idea what tools are available for denying access to web scrapers, this I should know given that I have built a few websites and know what to do to get pages serving quickly. Somehow I missed the memo on how to set your site up to not be scraped. Is there an nginx setting for that? This could be interesting for people that do scrape sites to know too, what basic reasonable measures can one take beyond lookin…
I'm sort of on the other side of this problem, but I think an interesting counter-measure came up in a recent Show HN[0].
IP bans alone are probably not enough to stop a motivated scraper, or work longterm, so checking whether or not traffic is coming from a cloud provider and throwing up a captcha would be a significant hurdle, while allowing humans on cloud-hosted VPNs to pass through without much trouble.
Re: How to scrape anything on the web and not get caught
#30What sorts of real-world and legitimate/ethical use cases are there for wholesale repeated scraping?