Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What do people use to prevent crawlers?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 123 posts

Re: Ask HN: What do people use to prevent crawlers?

#41

Post by FroshKiller »

huffmsa wrote: post #24

If you're getting a lot of crawler traffic, your site probably has information a lot of people find useful, so you should consider finding a way to monetize it. Otherwise, your best bet (hardest to get around in my experience) is monitoring for actual user I/O. Like if someone starts typing in an input field, real humans have to click on it beforehand, and most bots won't. Or if a user clicks next-page without the se…

You will create accessibility issues for users if you do this. The bias you'd encode in this idea of "human" behavior doesn't consider assistive software at all.

I don't click text inputs when my form-filling plugin enters my personal information on a payment screen. And even if I did, you wouldn't know it if I had JavaScript disabled.

Re: Ask HN: What do people use to prevent crawlers?

#42

Post by make3 »

huffmsa wrote: post #34
monodeldiablo wrote: post #28

"How do I stop all these dinner guests from eating this lovely pie I set out on the table?" I remember working hard on a project for a year, then releasing the data and visualizations online. I was very proud. It was very cool. Almost immediately, we saw grad students and research assistants across the globe scraping our site. I started brainstorming clever ways to fend off the scrapers with a colleague when my boss…

Side bar, if only Twitter would realize this and turn the firehose back on and charge $x per month for API access. They'd be profitable in a month.

They'd probably take a lot of flak from an ethical perspective though (scandal level). I suspect people would not be too excited to know Twitter straight up sells their content

Re: Ask HN: What do people use to prevent crawlers?

#43

Post by mkl »

wrath wrote: post #22

I've built crawlers that retrieve billions of web pages every month. We had a whole team working modifying the crawlers to resolve website changes, to reverse engineer ajax requests and solve complex problems like captcha solvers. Bottom line, if someone wants to crawl your website they will. What you can do, however, is make it hard so that the vast majority of developers can't do it (e.g. My tech crawl billions of…

> I've built crawlers that retrieve billions of web pages every month.

Wow, what were you doing with the data?

Re: Ask HN: What do people use to prevent crawlers?

#44

Post by 59nadir »

wrath wrote: post #22

I've built crawlers that retrieve billions of web pages every month. We had a whole team working modifying the crawlers to resolve website changes, to reverse engineer ajax requests and solve complex problems like captcha solvers. Bottom line, if someone wants to crawl your website they will. What you can do, however, is make it hard so that the vast majority of developers can't do it (e.g. My tech crawl billions of…

Do you feel bad at all about apparently making a business out of crawlers, but still apparently viewing it as bad enough that you want countermeasures against it? Don't you feel a slight bit hypocritical about this?

Re: Ask HN: What do people use to prevent crawlers?

#45

Post by tomascot »

mattm wrote: post #40

One thing I've thought about but never had the chance to put into practice would be to randomize CSS classes and IDs. Most web scraping relies on these to identify the content they are looking for. Imagine if everyday they changed? It would make things a lot more difficult. There would be disadvantages to actual users with this method like caching wouldn't work very well but maybe this alternative site could be displ…

It works. But you have to create empty elements, too. That way you also target code that is using xpath.

Also, you have to take into account regex.

Re: Ask HN: What do people use to prevent crawlers?

#46

Post by jorgec »

monodeldiablo wrote: post #28

"How do I stop all these dinner guests from eating this lovely pie I set out on the table?" I remember working hard on a project for a year, then releasing the data and visualizations online. I was very proud. It was very cool. Almost immediately, we saw grad students and research assistants across the globe scraping our site. I started brainstorming clever ways to fend off the scrapers with a colleague when my boss…

You are comparing two different things.

To have a free website is not the same than to unlimited grants such as a guy to taking the whole pie.

Or worst, lets say that you writes a free essay and everybody could read it and share. However, somebody takes it, deletes your name and put his name instead.

Re: Ask HN: What do people use to prevent crawlers?

#48

Post by Moru »

bmetz wrote: post #26

My favorite thing was to identify bots and instead of blocking them, switch to a slightly scrambled data set to make the scrape useless but look good to the developer who stole it. It was a ton of fun as a side project. I'd also suggest you add some innocent fake data to your real site and then set up google alerts of all of the above to catch traffic. About 50% of sites would respond positively to an email when you…

This is what we used to do. Then send a large zipfile with schreenshots and other data to the lawyers to handle the contact. Shortly after the scraping usually stopped. The contact and sell access wasnt an option because it was competitors taking the data.

Re: Ask HN: What do people use to prevent crawlers?

#49

Post by blowski »

huffmsa wrote: post #34
monodeldiablo wrote: post #28

"How do I stop all these dinner guests from eating this lovely pie I set out on the table?" I remember working hard on a project for a year, then releasing the data and visualizations online. I was very proud. It was very cool. Almost immediately, we saw grad students and research assistants across the globe scraping our site. I started brainstorming clever ways to fend off the scrapers with a colleague when my boss…

Side bar, if only Twitter would realize this and turn the firehose back on and charge $x per month for API access. They'd be profitable in a month.

Given the number of times people assert this, I think Twitter must have explored this option and decided it wasn't a viable option. I don't know why, but I'll assume that they're smart enough to have decided against it for a good reason.

Re: Ask HN: What do people use to prevent crawlers?

#50

Post by blowski »

monodeldiablo wrote: post #28

"How do I stop all these dinner guests from eating this lovely pie I set out on the table?" I remember working hard on a project for a year, then releasing the data and visualizations online. I was very proud. It was very cool. Almost immediately, we saw grad students and research assistants across the globe scraping our site. I started brainstorming clever ways to fend off the scrapers with a colleague when my boss…

I run a property website that lists properties for sale, rent, etc. A big part of my job is importing feeds, scraping sites (with permission) - and preventing others scraping our site. I know that some people will scrape, but I make I try to make it unprofitable for them to do so.

We do a bunch of other stuff, like adding fake properties so we can check who is scraping our content, and using tarpits.

Developers always argue with me that it's pissing in the wind, that "content wants to be free", that you shouldn't bother even trying to prevent it since it's inevitable. And yet it has helped. We did a split A/B test on preventing scrapers, and it turns out that it's quite effective.

Post reply on HN