Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What do people use to prevent crawlers?

news.ycombinator.com

Post reply on HN
Print view View original

91–100 of 123 posts

Re: Ask HN: What do people use to prevent crawlers?

#91

Post by viraptor »

digitalzombie wrote: post #39

You can use cloudfare but it's a small roadblock. I can still crawl that. Also you can do frontend rendering, it's a bit larger roadblock but you can use phantomJS or something to crawl that. IIRC there is a php framework that mutate your front end code but I'm not sure if it does it enough to stop a generalized xpath... Also I used to work for company where they employ people full time for crawling. It will even not…

Why do you think frontend rendering is harder? Every time I see it, I'm happy because there's a nice API it relies on - I can grab clean, structured data from it rather than trying to extract it from the text.

Re: Ask HN: What do people use to prevent crawlers?

#92

Post by [deleted] »

ketralnis wrote: post #82
wrath wrote: post #56

Earlier quoted context omitted.

Competitive intelligence. Crawling thousands of website, mashing up the data to analyze competitiveness between them, and selling it back. For example, cost of flights. Different websites provide different prices for the same flight. The technology crawls all the prices, combines the data, then resells it back to the websites. Everyone knows everyones prices, keeps competition high, lower prices for consumers.

Travel companies pay their GDS for every search they do. It costs so much that it's the primary cost centre for some of them. You were costing them thousands of dollars a day.

[deleted]

Re: Ask HN: What do people use to prevent crawlers?

#93

Post by wbl »

jerkstate wrote: post #64
monodeldiablo wrote: post #28

"How do I stop all these dinner guests from eating this lovely pie I set out on the table?" I remember working hard on a project for a year, then releasing the data and visualizations online. I was very proud. It was very cool. Almost immediately, we saw grad students and research assistants across the globe scraping our site. I started brainstorming clever ways to fend off the scrapers with a colleague when my boss…

You’re making a valid point for many cases, but there are definitely negative–value scrapers out there; let’s say you run a publishing platform and you see scrapers scraping your users content and then see that your site’s content has been rehosted for ad clicks. You can’t really license the content for this purpose and it’s bad for your brand and bad for your users.

That's what copyright lawsuits are for.

Re: Ask HN: What do people use to prevent crawlers?

#94

Post by RodericDay »

ketralnis wrote: post #82
wrath wrote: post #56

Earlier quoted context omitted.

Competitive intelligence. Crawling thousands of website, mashing up the data to analyze competitiveness between them, and selling it back. For example, cost of flights. Different websites provide different prices for the same flight. The technology crawls all the prices, combines the data, then resells it back to the websites. Everyone knows everyones prices, keeps competition high, lower prices for consumers.

Travel companies pay their GDS for every search they do. It costs so much that it's the primary cost centre for some of them. You were costing them thousands of dollars a day.

Good.

Re: Ask HN: What do people use to prevent crawlers?

#95

Post by bookmarkacc »

wrath wrote: post #22

I've built crawlers that retrieve billions of web pages every month. We had a whole team working modifying the crawlers to resolve website changes, to reverse engineer ajax requests and solve complex problems like captcha solvers. Bottom line, if someone wants to crawl your website they will. What you can do, however, is make it hard so that the vast majority of developers can't do it (e.g. My tech crawl billions of…

Very curious about this type of work. Is there is a good way to contact you to discuss this topic?

Re: Ask HN: What do people use to prevent crawlers?

#96

Post by flashman »

bmetz wrote: post #26

My favorite thing was to identify bots and instead of blocking them, switch to a slightly scrambled data set to make the scrape useless but look good to the developer who stole it. It was a ton of fun as a side project. I'd also suggest you add some innocent fake data to your real site and then set up google alerts of all of the above to catch traffic. About 50% of sites would respond positively to an email when you…

This is like the concept of a 'trap street': https://en.wikipedia.org/wiki/Trap_street

Re: Ask HN: What do people use to prevent crawlers?

#97

Post by sergiotapia »

ne0free wrote: post #69
wrath wrote: post #22

I've built crawlers that retrieve billions of web pages every month. We had a whole team working modifying the crawlers to resolve website changes, to reverse engineer ajax requests and solve complex problems like captcha solvers. Bottom line, if someone wants to crawl your website they will. What you can do, however, is make it hard so that the vast majority of developers can't do it (e.g. My tech crawl billions of…

How do you bypass google recaptcha

Mechanical turk

Re: Ask HN: What do people use to prevent crawlers?

#98

Post by laurieg »

Domain specific but if you detect a bot you can start giving it false information.

For example, a dictionary site. Someone tries to crawl your site after triggering your "This is a bot" code, serve bad data to every 20 requests. Mispell a word, Mislabel a noun as a verb, give an incorrect definition.

If you combine this with throttling then the value of scraping your site is greatly reduced. Also, most people won't come up with a super advanced crawler if they never get a "Permission denied, please stop crawling" message.

Re: Ask HN: What do people use to prevent crawlers?

#99

Post by DrScump »

wrath wrote: post #22

I've built crawlers that retrieve billions of web pages every month. We had a whole team working modifying the crawlers to resolve website changes, to reverse engineer ajax requests and solve complex problems like captcha solvers. Bottom line, if someone wants to crawl your website they will. What you can do, however, is make it hard so that the vast majority of developers can't do it (e.g. My tech crawl billions of…

If unwanted scraping can be distinguished from legitimate traffic, wouldn't a sort of honeypot strategy work such that you then provide those requests you've identified as likely to be unwelcome with fake or divergent data?

Re: Ask HN: What do people use to prevent crawlers?

#100

Post by 333c »

alecco wrote: post #81
wrath wrote: post #22

I've built crawlers that retrieve billions of web pages every month. We had a whole team working modifying the crawlers to resolve website changes, to reverse engineer ajax requests and solve complex problems like captcha solvers. Bottom line, if someone wants to crawl your website they will. What you can do, however, is make it hard so that the vast majority of developers can't do it (e.g. My tech crawl billions of…

Did you respect robots.txt?

It sounds to me like an obvious no, if they have a large team to get around countermeasures.
Post reply on HN