robots.txt then Zip bombs.
Why was this downvoted?
Ask HN: What do people use to prevent crawlers?
111–120 of 123 posts
Re: Ask HN: What do people use to prevent crawlers?
#112My favorite thing was to identify bots and instead of blocking them, switch to a slightly scrambled data set to make the scrape useless but look good to the developer who stole it. It was a ton of fun as a side project. I'd also suggest you add some innocent fake data to your real site and then set up google alerts of all of the above to catch traffic. About 50% of sites would respond positively to an email when you…
This is what we used to do. Then send a large zipfile with schreenshots and other data to the lawyers to handle the contact. Shortly after the scraping usually stopped. The contact and sell access wasnt an option because it was competitors taking the data.
And this is why it's an excellent idea to always scrape behind proxies. Never scrape from your own IP, or one easily traced to you.
Re: Ask HN: What do people use to prevent crawlers?
#113The other day I've made a Chrome extension for scrapping a protected website. It worked wonderfully, as it simulated a normal user session, bypassing the JavaScript protections the website has. You can also run such scripts with a headless browser for full automation, PhantomJS being an obvious choice. You really can't protect against this unless you start making the experience of regular visitors much worse.
Sure you can protect against this - there are several companies that use machine learning to spot small differences between selenium and real users (mouse delays etc). For example, it might detect that a mouse click is dispatched at exact intervals (and block it). To which you'd think "I'll just add Math.random() * 2000" which it'll easily detect as well. It's _definitely_ doable, but it's not as trivial as recording…
Yeah, but if you can use ML to pick out the differences between real users and bots then you can also use ML to mimic real user behavior. Both sides spend a lot of time and money trying to overcome each other, and end state is the same it was at the beginning: near total access to the data, at least for those sophisticated enough.
Re: Ask HN: What do people use to prevent crawlers?
#114If you're getting a lot of crawler traffic, your site probably has information a lot of people find useful, so you should consider finding a way to monetize it. Otherwise, your best bet (hardest to get around in my experience) is monitoring for actual user I/O. Like if someone starts typing in an input field, real humans have to click on it beforehand, and most bots won't. Or if a user clicks next-page without the se…
As someone who who does a fair amount of scraping for my job, it's always vastly preferred to simply pay for the data. It's generally more efficient and cost effective then scraping. Scraping is always the last resort: we need the data, it's not available for sale, and it's publicly accessible.
Re: Ask HN: What do people use to prevent crawlers?
#115Domain specific but if you detect a bot you can start giving it false information. For example, a dictionary site. Someone tries to crawl your site after triggering your "This is a bot" code, serve bad data to every 20 requests. Mispell a word, Mislabel a noun as a verb, give an incorrect definition. If you combine this with throttling then the value of scraping your site is greatly reduced. Also, most people won't c…
This is very true. My scraping efforts have become vastly more sophisticated after running into explicit attempts to block me. Now I've got all kinds of bells and whistles, and validate the data returned.
Re: Ask HN: What do people use to prevent crawlers?
#116If it's about content, SVG and convert all text to curves. /s
Can just screencap the rendered page and use OCR. If someone wants to scrape your website badly enough, they'll find a way.
Great idea. I'll keep this in my back pocket for future use.
The wiley and expensive way is to alter your page
Re: Ask HN: What do people use to prevent crawlers?
#117Earlier quoted context omitted.
You’re making a valid point for many cases, but there are definitely negative–value scrapers out there; let’s say you run a publishing platform and you see scrapers scraping your users content and then see that your site’s content has been rehosted for ad clicks. You can’t really license the content for this purpose and it’s bad for your brand and bad for your users.
That's what copyright lawsuits are for.
meh, I'll just block most scrapers and save 99% of my litigation costs to achieve the same result
Re: Ask HN: What do people use to prevent crawlers?
#118Scraping is my birthright, you'll never stop me.
Re: Ask HN: What do people use to prevent crawlers?
#119I've built crawlers that retrieve billions of web pages every month. We had a whole team working modifying the crawlers to resolve website changes, to reverse engineer ajax requests and solve complex problems like captcha solvers. Bottom line, if someone wants to crawl your website they will. What you can do, however, is make it hard so that the vast majority of developers can't do it (e.g. My tech crawl billions of…
How do you prevent your website from not functioning over time for legitimate users, though? I'm a Sysadmin & not a coder or developer, so the tricks you can do are a little foreign to me. Can you provide examples? Why don't Adidas/Nike/et. al. do this to fight the likes of sneakerbots?
Re: Ask HN: What do people use to prevent crawlers?
#120Earlier quoted context omitted.
Competitive intelligence. Crawling thousands of website, mashing up the data to analyze competitiveness between them, and selling it back. For example, cost of flights. Different websites provide different prices for the same flight. The technology crawls all the prices, combines the data, then resells it back to the websites. Everyone knows everyones prices, keeps competition high, lower prices for consumers.
Travel companies pay their GDS for every search they do. It costs so much that it's the primary cost centre for some of them. You were costing them thousands of dollars a day.
GDS?