Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Should you block web scrapers?

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 13 posts

Ask HN: Should you block web scrapers?

#1

Post by jorgecurio »

Let's leave the ethics debate aside and ask this question: is it realistic to invest time and money into blocking scrapers?

I see distill networks is popular, but what is their pricing? try to scrape crunchbase and see what happens. It's actually very good I tested with phantomjs, selenium + firefox and it's able to detect and successfully block the traffic.

I'm willing to invest time and money into preventing competitors from launching based on scraping my data that I worked hard for.

Is this something that I can implement? Sounds complicated in two parts

1) building fingerprints on user agent and other information. What do you include, where do you start?

2) Distinguishing malicious requests on a time series while avoiding false positives (some people compulsively click next button twice if it doesn't load immediately

What are some surefire ways that will give me high degree of confidence that scraper will fail.

Re: Ask HN: Should you block web scrapers?

#2

Post by softwaredev__ »

You should go after the low hanging fruits first. Check if the request is coming from a trusted browser (Chrome, Firefox, Safari, etc). If not, block them. I'm sure that'll cut down scrapes by a lot. And then, you can go after some behavioral stuff. You can do things like measure the milliseconds between requests. If it's too fast for a human to have clicked that link after the initial page load, then you know it's a bot.

Re: Ask HN: Should you block web scrapers?

#3

Post by ldjb »

softwaredev__ wrote: post #2

You should go after the low hanging fruits first. Check if the request is coming from a trusted browser (Chrome, Firefox, Safari, etc). If not, block them. I'm sure that'll cut down scrapes by a lot. And then, you can go after some behavioral stuff. You can do things like measure the milliseconds between requests. If it's too fast for a human to have clicked that link after the initial page load, then you know it's a…

Please don't do that. Some of us use rather obscure web browsers, and unless you keep the whitelist updated, users of newer browsers will be locked out of the website.

Besides, if your website blocks me, I'm not going to switch browser — I'll just go elsewhere.

Re: Ask HN: Should you block web scrapers?

#4

Post by ldjb »

There is no way to completely block scrapers. The most you can do is make life difficult for those writing them. You also have to ensure you're not blocking genuine users.

Instead of investing time and money trying to prevent competitors, why not use that time and money to innovate on the service you provide so that there will be no point in setting up a competitor?

Re: Ask HN: Should you block web scrapers?

#5

Post by jorgecurio »

softwaredev__ wrote: post #2

You should go after the low hanging fruits first. Check if the request is coming from a trusted browser (Chrome, Firefox, Safari, etc). If not, block them. I'm sure that'll cut down scrapes by a lot. And then, you can go after some behavioral stuff. You can do things like measure the milliseconds between requests. If it's too fast for a human to have clicked that link after the initial page load, then you know it's a…

what worries me is false positives preventing a user from accessing the website

Re: Ask HN: Should you block web scrapers?

#6

Post by jorgecurio »

ldjb wrote: post #4

There is no way to completely block scrapers. The most you can do is make life difficult for those writing them. You also have to ensure you're not blocking genuine users. Instead of investing time and money trying to prevent competitors, why not use that time and money to innovate on the service you provide so that there will be no point in setting up a competitor?

Of course it's not possible to 100% block ALL scrapers, much like the debate surrounding obfusticating javascript.

The point is to increase the cost of developing a scraper to a maxima where ROI begins to rapidly deteriorate. I'm trying to identify techniques and methods which will:

- identify real users apart from bot users

- increase the costs for the bot

Re: Ask HN: Should you block web scrapers?

#7

Post by softwaredev__ »

ldjb wrote: post #3
softwaredev__ wrote: post #2

You should go after the low hanging fruits first. Check if the request is coming from a trusted browser (Chrome, Firefox, Safari, etc). If not, block them. I'm sure that'll cut down scrapes by a lot. And then, you can go after some behavioral stuff. You can do things like measure the milliseconds between requests. If it's too fast for a human to have clicked that link after the initial page load, then you know it's a…

Please don't do that. Some of us use rather obscure web browsers, and unless you keep the whitelist updated, users of newer browsers will be locked out of the website. Besides, if your website blocks me, I'm not going to switch browser — I'll just go elsewhere.

Just curious, what obscure browsers do you use?

Re: Ask HN: Should you block web scrapers?

#8

Post by ldjb »

softwaredev__ wrote: post #7
ldjb wrote: post #3

Earlier quoted context omitted.

Please don't do that. Some of us use rather obscure web browsers, and unless you keep the whitelist updated, users of newer browsers will be locked out of the website. Besides, if your website blocks me, I'm not going to switch browser — I'll just go elsewhere.

Just curious, what obscure browsers do you use?

Some examples are Lynx, Iceweasel, Chromium, Steam's in-game browser and the Nintendo 3DS browser. Okay, none of them are that obscure but they probably aren't the first things that spring to mind when you think of browsers.

Depending on your method of blocking browsers, it might automatically lump Iceweasel together with Firefox and Chromium together with Chrome. But if this is something you really want to do, you have to be very careful about these things.

Re: Ask HN: Should you block web scrapers?

#10

Post by jorgecurio »

ldjb wrote: post #8
softwaredev__ wrote: post #7

Earlier quoted context omitted.

Just curious, what obscure browsers do you use?

Some examples are Lynx, Iceweasel, Chromium, Steam's in-game browser and the Nintendo 3DS browser. Okay, none of them are that obscure but they probably aren't the first things that spring to mind when you think of browsers. Depending on your method of blocking browsers, it might automatically lump Iceweasel together with Firefox and Chromium together with Chrome. But if this is something you really want to do, you h…

so it seems like the only sure-fire way of detecting bad bots is via an IP address and not relying on Browser Agent.

Is it any beneficial if there's a way to detect if a given IP address is bad or not right from the get go? Possibly by searching through millions of IP addresses that are blacklisted through a bloom filter and REST API? Do you think this would help-> a long roster of abusive IP address databases?

Post reply on HN