How to scrape anything on the web and not get caught
tinyendian.com
How to scrape anything on the web and not get caught
1–10 of 57 posts
Re: How to scrape anything on the web and not get caught
#2I don't know if scrapy handles this, but I've run into issues with sites fingerprinting my browser. Proxies help, but there are other ways to identify site visitors aside from IP addresses.
Re: How to scrape anything on the web and not get caught
#3>after introducing proxies my crawl times grew by an order of magnitude from minutes to hours
Yeah, same experience. Right now I use luminati.io datacenter IPs that work ok, anyone know of a cheaper option that works well? Scraping tens of millions of pages a month.
Re: How to scrape anything on the web and not get caught
#4I don't know if scrapy handles this, but I've run into issues with sites fingerprinting my browser. Proxies help, but there are other ways to identify site visitors aside from IP addresses.
Rotating user agents ?
Re: How to scrape anything on the web and not get caught
#5I don't know if scrapy handles this, but I've run into issues with sites fingerprinting my browser. Proxies help, but there are other ways to identify site visitors aside from IP addresses.
Some proxies will also include your IP address in the form of an "X-Forwarded-For" header.
Re: How to scrape anything on the web and not get caught
#6>after introducing proxies my crawl times grew by an order of magnitude from minutes to hours Yeah, same experience. Right now I use luminati.io datacenter IPs that work ok, anyone know of a cheaper option that works well? Scraping tens of millions of pages a month.
You could use https://oxylabs.io/ or buy some regular VPN accounts and build a http proxy wrapper around that. Extremely cheap and works well. There’s a lot of existing projects on Github for that too.
Re: How to scrape anything on the web and not get caught
#7Careful. Using open proxies could be considered unauthorized access in some jurisdictions. Some of these proxies were installed without the user's permission.
This is my favorite consensual alternative: https://github.com/mattes/rotating-proxy
Re: How to scrape anything on the web and not get caught
#8>after introducing proxies my crawl times grew by an order of magnitude from minutes to hours Yeah, same experience. Right now I use luminati.io datacenter IPs that work ok, anyone know of a cheaper option that works well? Scraping tens of millions of pages a month.
You could use https://oxylabs.io/ or buy some regular VPN accounts and build a http proxy wrapper around that. Extremely cheap and works well. There’s a lot of existing projects on Github for that too.
oxylabs.io seems more expensive than luminati, minimum of $178/month. Not clear if they charge for bandwidth.
Problem with VPN is it's shared and hard to get a lot of IPs, any specific ones I could get say 100 dedicated US IPs for a reasonable price?
Re: How to scrape anything on the web and not get caught
#9I have no idea what tools are available for denying access to web scrapers, this I should know given that I have built a few websites and know what to do to get pages serving quickly. Somehow I missed the memo on how to set your site up to not be scraped. Is there an nginx setting for that?
This could be interesting for people that do scrape sites to know too, what basic reasonable measures can one take beyond looking at logs and doing IP bans?
Re: How to scrape anything on the web and not get caught
#10Careful. Using open proxies could be considered unauthorized access in some jurisdictions. Some of these proxies were installed without the user's permission. This is my favorite consensual alternative: https://github.com/mattes/rotating-proxy
This solution will not work with HTTPS. There are other alternatives and it's easy to roll your own.