Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to scrape anything on the web and not get caught

tinyendian.com

Post reply on HN
Print view View original

11–20 of 57 posts

Re: How to scrape anything on the web and not get caught

#11

Post by pjc50 »

Theodores wrote: post #9

I have no idea what tools are available for denying access to web scrapers, this I should know given that I have built a few websites and know what to do to get pages serving quickly. Somehow I missed the memo on how to set your site up to not be scraped. Is there an nginx setting for that? This could be interesting for people that do scrape sites to know too, what basic reasonable measures can one take beyond lookin…

The cheap technique is rate limiting, but there are also "canary pages" (linked invisibly but should not be accessed by humans) and other techniques for looking to see if the flow through the site is as expected.

Re: How to scrape anything on the web and not get caught

#12

Post by tomarr »

ikeboy wrote: post #3

>after introducing proxies my crawl times grew by an order of magnitude from minutes to hours Yeah, same experience. Right now I use luminati.io datacenter IPs that work ok, anyone know of a cheaper option that works well? Scraping tens of millions of pages a month.

I suppose the economics of it comes in to play in a similar vein to mailchimp, the lower the pricing the more scammier clients and the more IPs they lose to blacklists.

Re: How to scrape anything on the web and not get caught

#13

Post by dredmorbius »

JeanMarcS wrote: post #4
sram1337 wrote: post #2

I don't know if scrapy handles this, but I've run into issues with sites fingerprinting my browser. Proxies help, but there are other ways to identify site visitors aside from IP addresses.

Rotating user agents ?

Minimally effective, or outright detrimental, in my own limited testing.

Re: How to scrape anything on the web and not get caught

#14

Post by ikeboy »

tomarr wrote: post #12
ikeboy wrote: post #3

>after introducing proxies my crawl times grew by an order of magnitude from minutes to hours Yeah, same experience. Right now I use luminati.io datacenter IPs that work ok, anyone know of a cheaper option that works well? Scraping tens of millions of pages a month.

I suppose the economics of it comes in to play in a similar vein to mailchimp, the lower the pricing the more scammier clients and the more IPs they lose to blacklists.

Not really. Mail is default bad, you need to build up trust just to get a tiny amount of deliverability. Fetching webpages is default good until you're detected as bad.

The thing is, a lot of scraping goes unnoticed. Maybe you get an extra thousand hits here and there. But every spam campaign gets noticed and results in some percentage of spam complaints from users.

Re: How to scrape anything on the web and not get caught

#15

Post by dewey »

ikeboy wrote: post #8
dewey wrote: post #6

Earlier quoted context omitted.

You could use https://oxylabs.io/ or buy some regular VPN accounts and build a http proxy wrapper around that. Extremely cheap and works well. There’s a lot of existing projects on Github for that too.

oxylabs.io seems more expensive than luminati, minimum of $178/month. Not clear if they charge for bandwidth. Problem with VPN is it's shared and hard to get a lot of IPs, any specific ones I could get say 100 dedicated US IPs for a reasonable price?

Depends on your scale and what you negotiate. The shared nature of VPN is usually not a huge problem but depends on your use case. Most VPN providers have a better deal for bigger customers so you can just buy multiple accounts in bulk with each having for example 5 connections and use that. For US specifically it often happens that they have 100s of servers and vpn configs so you can build something out of that.

Re: How to scrape anything on the web and not get caught

#16

Post by jstarfish »

Theodores wrote: post #9

I have no idea what tools are available for denying access to web scrapers, this I should know given that I have built a few websites and know what to do to get pages serving quickly. Somehow I missed the memo on how to set your site up to not be scraped. Is there an nginx setting for that? This could be interesting for people that do scrape sites to know too, what basic reasonable measures can one take beyond lookin…

IPs are naive and too broad. Force users to authenticate to see your content. Unauthenticated viewers get to see only a scrambled version. Now you can track usage by username.

Establish a baseline as to what constitutes "normal" browsing.

Any user exceeding a threshold of page requests gets rate limited/banned.

Stopping people from recording public information is unrealistic, but you can certainly make it more of a pain in the ass.

Re: How to scrape anything on the web and not get caught

#18

Post by endisukaj »

darpa_escapee wrote: post #10
buildbuildbuild wrote: post #7

Careful. Using open proxies could be considered unauthorized access in some jurisdictions. Some of these proxies were installed without the user's permission. This is my favorite consensual alternative: https://github.com/mattes/rotating-proxy

This solution will not work with HTTPS. There are other alternatives and it's easy to roll your own.

For example?

Re: How to scrape anything on the web and not get caught

#19

Post by dingaling »

sram1337 wrote: post #2

I don't know if scrapy handles this, but I've run into issues with sites fingerprinting my browser. Proxies help, but there are other ways to identify site visitors aside from IP addresses.

Can't you use a service like Panopticlick to 'tune' the browser fingerprint to a majority percentile? Particularly if running in clean VM images.
Post reply on HN