Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What do people use to prevent crawlers?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 123 posts

Re: Ask HN: What do people use to prevent crawlers?

#61

Post by calafrax »

monodeldiablo wrote: post #28

"How do I stop all these dinner guests from eating this lovely pie I set out on the table?" I remember working hard on a project for a year, then releasing the data and visualizations online. I was very proud. It was very cool. Almost immediately, we saw grad students and research assistants across the globe scraping our site. I started brainstorming clever ways to fend off the scrapers with a colleague when my boss…

That is good advice from a technical standpoint but from a legal standpoint creating security features that prevent scraping gives you a clearer cause of action against scrapers so if someone starts making a lot of money off your content you get leverage to force them to pay for it.

Re: Ask HN: What do people use to prevent crawlers?

#62

Post by amelius »

alexcnwy wrote: post #15
pmx wrote: post #14

Earlier quoted context omitted.

They generally are fairly slow but it doesn't matter because you just set it and leave it to it until it has all the data you want, leaving you to get on with other things.

Plus very easy to parallelize

Unless the websites work together to protect themselves against crawlers that work that way (e.g., rejecting requests to different hosts from the same origin within a time interval).

Re: Ask HN: What do people use to prevent crawlers?

#63

Post by majewsky »

pbhjpbhj wrote: post #25
alexcnwy wrote: post #16

Earlier quoted context omitted.

Can just screencap the rendered page and use OCR. If someone wants to scrape your website badly enough, they'll find a way.

Use a really bad font with terrible kerning and very similar 1lI and oO0, etc.. /s

Spellcheckers and error-correcting OCR are a thing. In fact, spellchckers are so well-understood that they're a frequent assignment for CS undergraduates.

Re: Ask HN: What do people use to prevent crawlers?

#64

Post by jerkstate »

monodeldiablo wrote: post #28

"How do I stop all these dinner guests from eating this lovely pie I set out on the table?" I remember working hard on a project for a year, then releasing the data and visualizations online. I was very proud. It was very cool. Almost immediately, we saw grad students and research assistants across the globe scraping our site. I started brainstorming clever ways to fend off the scrapers with a colleague when my boss…

You’re making a valid point for many cases, but there are definitely negative–value scrapers out there; let’s say you run a publishing platform and you see scrapers scraping your users content and then see that your site’s content has been rehosted for ad clicks. You can’t really license the content for this purpose and it’s bad for your brand and bad for your users.

Re: Ask HN: What do people use to prevent crawlers?

#66

Post by calafrax »

There are a variety of methods that can be deployed:

1) request fingerprinting - browser request headers have arbitrary patterns that depend on user agent. matching user agent strings with a database of the request header fingerprints allows you to filter out anyone who is not using a real browser who hasn't taken the time to correctly spoof the headers. this will filter out low skill low energy scrapers and create higher costs.

2) put javascript in the page that tracks mouse movement and pings back. this forces scrapers to simulate mouse movement in a js execution environment or reverse engineer your ping back script. this is a very high hurdle and once again forces much more computationally intensive scraping and also much more sophisticated engineering effort.

3) do access pattern detection. require valid refer headers. don't allow api access without page access, etc. check that assets from page are loaded. etc.

4) use maxmind database and treat as suspicious any access not from a consumer isp. block access from aws, gcp, azure, and other cloud services offering cheap ip rental.

Re: Ask HN: What do people use to prevent crawlers?

#67

Post by calafrax »

ThePhysicist wrote: post #6

I assume you are concerned about crawlers that do not respect the robots.txt file (which is the polite way to restrict them from indexing your side, but does not provide any actual protection if crawlers chose to ignore the file). Cloudflare has a tool for doing this (now part of their core service): https://blog.cloudflare.com/introducing-scrapeshield-discove... There's a nice Github repo with some advice on blockin…

> CAPTCHA solution is the most robust one captcha solving services are very cheap so captchas are basically useless for anything of value.

Re: Ask HN: What do people use to prevent crawlers?

#68

Post by Waterluvian »

monodeldiablo wrote: post #28

"How do I stop all these dinner guests from eating this lovely pie I set out on the table?" I remember working hard on a project for a year, then releasing the data and visualizations online. I was very proud. It was very cool. Almost immediately, we saw grad students and research assistants across the globe scraping our site. I started brainstorming clever ways to fend off the scrapers with a colleague when my boss…

That's a great point. I'm distracted a bit by how you built a project for a year without understanding what the business strategy was going to be. Some big time comms breakdown there eh?

Re: Ask HN: What do people use to prevent crawlers?

#69

Post by ne0free »

wrath wrote: post #22

I've built crawlers that retrieve billions of web pages every month. We had a whole team working modifying the crawlers to resolve website changes, to reverse engineer ajax requests and solve complex problems like captcha solvers. Bottom line, if someone wants to crawl your website they will. What you can do, however, is make it hard so that the vast majority of developers can't do it (e.g. My tech crawl billions of…

How do you bypass google recaptcha

Re: Ask HN: What do people use to prevent crawlers?

#70

Post by Rjevski »

Don't prevent them. The same data you let humans access for free should be accessible via bots. If you only want to give out a "reasonable" amount of data, that humans wouldn't usually exceed but bots would, then define a rate-limit that wouldn't inconvenience humans and then apply it for everyone - bot or not. That way you're discriminating based on the amount of data instead of whether it's a bot or not. It will thwart people simply paying humans to scrape the data (which would happen if you magically found a way to block bots) while not inconveniencing humans who use a bot to make their job easier while scraping a reasonable amount of data.
Post reply on HN