Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bypassing website anti-scraping protections

kb.apify.com

Post reply on HN
Print view View original

41–50 of 121 posts

Re: Bypassing website anti-scraping protections

#41

Post by zzzcpan »

dvfjsdhgfv wrote: post #40

> there are already anti-scraping solutions on the market that can detect its usage based on a variable it puts into the browser's window.navigator property. Thankfully, we have developed a solution which removes the property from the web browser Does anyone know what exactly the property in question is?

Probably navigator.webdriver, but there are multiple properties.

https://antoinevastel.com/bot%20detection/2017/08/05/detect-...

https://antoinevastel.com/bot%20detection/2018/01/17/detect-...

Re: Bypassing website anti-scraping protections

#42

Post by dvfjsdhgfv »

zzzcpan wrote: post #41
dvfjsdhgfv wrote: post #40

> there are already anti-scraping solutions on the market that can detect its usage based on a variable it puts into the browser's window.navigator property. Thankfully, we have developed a solution which removes the property from the web browser Does anyone know what exactly the property in question is?

Probably navigator.webdriver, but there are multiple properties. https://antoinevastel.com/bot%20detection/2017/08/05/detect-... https://antoinevastel.com/bot%20detection/2018/01/17/detect-...

That's why I was wondering. Last time I checked headless Chrome could be pretty reliably detected in a number of ways, as you say. That they mention just one variable seems quite odd, given that they position themselves as specialists in the field.

Re: Bypassing website anti-scraping protections

#43

Post by jardah »

dvfjsdhgfv wrote: post #42
zzzcpan wrote: post #41

Earlier quoted context omitted.

Probably navigator.webdriver, but there are multiple properties. https://antoinevastel.com/bot%20detection/2017/08/05/detect-... https://antoinevastel.com/bot%20detection/2018/01/17/detect-...

That's why I was wondering. Last time I checked headless Chrome could be pretty reliably detected in a number of ways, as you say. That they mention just one variable seems quite odd, given that they position themselves as specialists in the field.

The webdriver property is as far as we know the only one that stays different if you use non-headless chrome with puppeteer. Rest can be handled by use of non-headless chrome as mentioned in the article.

But you are right, after reading through it again, this section of the article should be improved.

Re: Bypassing website anti-scraping protections

#44

Post by greglindahl »

userbinator wrote: post #3

For example, for google.com, you can typically make only around 300 requests per day, and if you reach this limit, you will see a CAPTCHA instead of search results. 300 is pretty easy to achieve if you're "Googling hard enough" (make 5 slightly different queries, go through the 20 pages of results it's willing to show you, repeat 3 times...), and I've seen it trigger far before that if you are searching for more obsc…

From my experience at blekko, 99.9% of the "people" who go deep into the results pages for a single query are actually bots. You're a very unusual user, and there are a lot of bots.

Re: Bypassing website anti-scraping protections

#45

Post by textmode »

0x00000000 wrote: post #23
userbinator wrote: post #3

For example, for google.com, you can typically make only around 300 requests per day, and if you reach this limit, you will see a CAPTCHA instead of search results. 300 is pretty easy to achieve if you're "Googling hard enough" (make 5 slightly different queries, go through the 20 pages of results it's willing to show you, repeat 3 times...), and I've seen it trigger far before that if you are searching for more obsc…

Instagram is the worst I have come across. If you are on a page with 1000+ pictures trying to find something near the bottom, you have to let it load each new group sequentially, then after a while it starts timing you out for like 60 seconds or longer every couple times you load more. God forbid you accidentally navigate away while scrolling you have to start all over again from the top. Due to recent events it seem…

This sounds like an issue that is specific to Javascript-controlled browsers. If using a traditional, non-Javascript tcp/tls/http client it is trivial to extract the image urls and other information from the page using a single HTTP request (and from each successive page using more HTTP requests in a single connection, if "has_next_page" is "true"). No "API" needed. Can you provide an example of a single page with 1000+ images?

Re: Bypassing website anti-scraping protections

#46

Post by dredmorbius »

learc83 wrote: post #31

A company I consulted for was using a paid API to handle search. Despite the fact that the entire site was available in an easy to scrape XML format, scrapers kept using the search feature. They were trying very hard to overcome my countermeasures--they had a seemingly limitless pool of IPs, they were rotating user agent strings, and they tried to randomize search behavior. Everytime I implemented a new countermeasur…

An explicit message to use that feed as part of the countermeasures might be useful. Did/do you do this?

Re: Bypassing website anti-scraping protections

#48

Post by vbezhenar »

erikrothoff wrote: post #47

I was playing around with the idea of using Tor to get around IP blocks. I played around a bit with code but the Tor binary dependency was a bit much for my use case. Curious to know if anyone else tried this?

I don't like those attempts. Tor is easily detectable and those attempts just make Tor network banned from website, hurting legitimate Tor users.

Re: Bypassing website anti-scraping protections

#49

Post by always_good »

erikrothoff wrote: post #47

I was playing around with the idea of using Tor to get around IP blocks. I played around a bit with code but the Tor binary dependency was a bit much for my use case. Curious to know if anyone else tried this?

Everyone else has the same idea which is why it often makes sense to block Tor outright.

Re: Bypassing website anti-scraping protections

#50

Post by aruggirello »

zzzcpan wrote: post #41
dvfjsdhgfv wrote: post #40

> there are already anti-scraping solutions on the market that can detect its usage based on a variable it puts into the browser's window.navigator property. Thankfully, we have developed a solution which removes the property from the web browser Does anyone know what exactly the property in question is?

Probably navigator.webdriver, but there are multiple properties. https://antoinevastel.com/bot%20detection/2017/08/05/detect-... https://antoinevastel.com/bot%20detection/2018/01/17/detect-...

Is there anything like this to reliably detect Firefox headless?
Post reply on HN