Bypassing website anti-scraping protections

Post by eboyjr » Sat, Jun 02, 2018, 3:19 AM UTC

    if (navigator.webdriver || (Navigator.prototype && Object.getOwnPropertyDescriptors(Navigator.prototype)["webdriver"])) {
        // Chrome headless detected - navigator.webdriver exists or was redefined
    }

Post by jardah » Sat, Jun 02, 2018, 5:20 AM UTC

eboyjr wrote: post #101

> we have developed a solution which removes the property from the web browser and thus prevents these kind of protections from figuring out that the browser is automated Eli Grey and I have bypassed your "hideWebDriver()" function[1] in a single line of code: if (navigator.webdriver || (Navigator.prototype && Object.getOwnPropertyDescriptors(Navigator.prototype)["webdriver"])) { // Chrome headless detected - navigat…

Post by textmode » Sat, Jun 02, 2018, 6:29 AM UTC

0x00000000 wrote: post #84

textmode wrote: post #45

Earlier quoted context omitted.

This sounds like an issue that is specific to Javascript-controlled browsers. If using a traditional, non-Javascript tcp/tls/http client it is trivial to extract the image urls and other information from the page using a single HTTP request (and from each successive page using more HTTP requests in a single connection, if "has_next_page" is "true"). No "API" needed. Can you provide an example of a single page with 10…

https://www.instagram.com/ryuji513 It looks like it just hits https://www.instagram.com/graphql/query/.. . every time you scroll down so if you scroll too fast it just hammers it and throttles your requests to that endpoint.

Post by icebraining » Sat, Jun 02, 2018, 9:31 AM UTC

DFHippie wrote: post #38

icebraining wrote: post #32

Earlier quoted context omitted.

Eh, the principle might be good (though it's not odd that not everyone shares the same principles), but one can hold it and still have exceptions. For example, what about a governmental institution or a public company¹? What about a semi-public company, like a monopolist utility? What if the uploader of the data is OK with it, but the site hoster prevents it? ¹ in the sense of owned by the State, not listed on the st…

I'm not saying there aren't legitimate reasons for writing scrapers. I've written plenty myself. It was just odd to see this disregarded entirely. As for the commonality of principles, game theory explains most of them, so it isn't more surprising than that we all work with the same prime numbers, say. A simple principle of reciprocity will produce something along the lines of "respect other people's wishes".

Post by jorge_gonzalez » Sat, Jun 02, 2018, 9:55 AM UTC

rosha wrote: post #106

Scraping on high volumes any serp is pain if your business relies on it and most services out there do not work on high volumes or they work and crazy expensive. I have checked few solutions out there, I am using now proxycrawl. Developers of their api helped me get a very high volume of Serp data from different search engines like Yandex, google and yahoo and bing. I also use them for Javascript crawling as for our…