11–20 of 123 posts
The other day I've made a Chrome extension for scrapping a protected website. It worked wonderfully, as it simulated a normal user session, bypassing the JavaScript protections the website has. You can also run such scripts with a headless browser for full automation, PhantomJS being an obvious choice. You really can't protect against this unless you start making the experience of regular visitors much worse.
The other day I've made a Chrome extension for scrapping a protected website. It worked wonderfully, as it simulated a normal user session, bypassing the JavaScript protections the website has. You can also run such scripts with a headless browser for full automation, PhantomJS being an obvious choice. You really can't protect against this unless you start making the experience of regular visitors much worse.
But your crawler must be very slow then.
If it's about content, SVG and convert all text to curves. /s
If someone wants to scrape your website badly enough, they'll find a way.
Most crawlers will make hundreds of requests in five minutes, while legitimate viewers will make be bellow 100.
Both worked, both worked well with http downloads and selenium (and common techniques). Neither worked against someone dedicated enough - but there are the usual tricks for bypassing them (which we used, to test our own stuff).
We also developed something in-house, but that never helps.
The other day I've made a Chrome extension for scrapping a protected website. It worked wonderfully, as it simulated a normal user session, bypassing the JavaScript protections the website has. You can also run such scripts with a headless browser for full automation, PhantomJS being an obvious choice. You really can't protect against this unless you start making the experience of regular visitors much worse.
For example, it might detect that a mouse click is dispatched at exact intervals (and block it). To which you'd think "I'll just add Math.random() * 2000" which it'll easily detect as well.
It's _definitely_ doable, but it's not as trivial as recording a selenium macro. (Not to mention these tools look for selenium presence and extensions anyway).
I run a cronjob every 5 minutes that parses httpd access log. If there's an IP with abnormally large request number, it blocks it. Most crawlers will make hundreds of requests in five minutes, while legitimate viewers will make be bellow 100.
This is pretty cheap to do and I've seen it done before in several places I've worked (on the defending side).