Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What do people use to prevent crawlers?

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 123 posts

Re: Ask HN: What do people use to prevent crawlers?

#71

Post by grogenaut »

huffmsa wrote: post #37
danpalmer wrote: post #36

Earlier quoted context omitted.

Twitter do charge for the firehose, and I hear it’s a reasonable amount of revenue. A person can’t go and buy it on a credit card, but there are authorised sellers, enterprise sales people, etc. Bear in mind that it’s a large technical feat to be able to ingest the firehose effectively, so it’s not really suitable for consumers.

I mean sell it piecemeal. Add a butt ton of endpoints. Make it easy to use, easy to integrate. Sell it on credit card.

What do you mean piecemeal? Like random subset? Go answered why not the whole thing already.

Re: Ask HN: What do people use to prevent crawlers?

#72

Post by anonymoushn »

danpalmer wrote: post #36
huffmsa wrote: post #34

Earlier quoted context omitted.

Side bar, if only Twitter would realize this and turn the firehose back on and charge $x per month for API access. They'd be profitable in a month.

Twitter do charge for the firehose, and I hear it’s a reasonable amount of revenue. A person can’t go and buy it on a credit card, but there are authorised sellers, enterprise sales people, etc. Bear in mind that it’s a large technical feat to be able to ingest the firehose effectively, so it’s not really suitable for consumers.

Do you know what the current bitrate is? In 2011 I looked into buying this but gave up because it was too hard to buy. Twitter was pretending at the time that it was hard to process so much data, but it was incredibly easy as long as you don't do it in Ruby.

Re: Ask HN: What do people use to prevent crawlers?

#73

Post by grogenaut »

Moru wrote: post #48
bmetz wrote: post #26

My favorite thing was to identify bots and instead of blocking them, switch to a slightly scrambled data set to make the scrape useless but look good to the developer who stole it. It was a ton of fun as a side project. I'd also suggest you add some innocent fake data to your real site and then set up google alerts of all of the above to catch traffic. About 50% of sites would respond positively to an email when you…

This is what we used to do. Then send a large zipfile with schreenshots and other data to the lawyers to handle the contact. Shortly after the scraping usually stopped. The contact and sell access wasnt an option because it was competitors taking the data.

I did some scraping for a lawyer back in like 01 from other lawyers. He got a c&d and told me to turn it off (we were done anyway).

Funny part was the lawyer on the other side wanted us to return all of the content on disk. Not show what we had copied but literally return it. My lawyer laughed about it. The other lawyer was smart/savvy enough to be effectively using the internet in 01 but didn't really understand the tech.

Other funny part is if he had generalized his site outside of law he would have had a major business these days.

Re: Ask HN: What do people use to prevent crawlers?

#75

Post by danpalmer »

anonymoushn wrote: post #72
danpalmer wrote: post #36

Earlier quoted context omitted.

Twitter do charge for the firehose, and I hear it’s a reasonable amount of revenue. A person can’t go and buy it on a credit card, but there are authorised sellers, enterprise sales people, etc. Bear in mind that it’s a large technical feat to be able to ingest the firehose effectively, so it’s not really suitable for consumers.

Do you know what the current bitrate is? In 2011 I looked into buying this but gave up because it was too hard to buy. Twitter was pretending at the time that it was hard to process so much data, but it was incredibly easy as long as you don't do it in Ruby.

I don't know, but I imagine part of the reason for using resellers now is probably support contracts and stuff like that which go along with it. I'd imagine they also sell custom firehoses that pre-filter certain data, etc.

Re: Ask HN: What do people use to prevent crawlers?

#76

Post by prathiks »

inglor wrote: post #18

We've used Incapsula (cheap and works, but awful support and service) and Distil (expensive and works, great support but steep pricing). Both worked, both worked well with http downloads and selenium (and common techniques). Neither worked against someone dedicated enough - but there are the usual tricks for bypassing them (which we used, to test our own stuff). We also developed something in-house, but that never he…

What do you mean by dedicated enough?? I guess the cost of attack would be too much in that case.

Re: Ask HN: What do people use to prevent crawlers?

#78

Post by Veratyr »

danpalmer wrote: post #36
huffmsa wrote: post #34

Earlier quoted context omitted.

Side bar, if only Twitter would realize this and turn the firehose back on and charge $x per month for API access. They'd be profitable in a month.

Twitter do charge for the firehose, and I hear it’s a reasonable amount of revenue. A person can’t go and buy it on a credit card, but there are authorised sellers, enterprise sales people, etc. Bear in mind that it’s a large technical feat to be able to ingest the firehose effectively, so it’s not really suitable for consumers.

> but there are authorised sellers

I don't believe this is true. It used to be the case but Twitter went and put all API sales under Gnip.

Re: Ask HN: What do people use to prevent crawlers?

#79

Post by wrath »

ne0free wrote: post #69
wrath wrote: post #22

I've built crawlers that retrieve billions of web pages every month. We had a whole team working modifying the crawlers to resolve website changes, to reverse engineer ajax requests and solve complex problems like captcha solvers. Bottom line, if someone wants to crawl your website they will. What you can do, however, is make it hard so that the vast majority of developers can't do it (e.g. My tech crawl billions of…

How do you bypass google recaptcha

I can't provide details on any innovations we've done with sites like google, but in general if you want to crawl google you'll want to get "many, many" IP addresses. I've heard of people using services like 2captcha.com but the best way is to obfuscate who you are.

If you can hit Google 60 times per minute per IP before getting blocked and you need to crawl them 1000 times per minute, you need 17 IPs per hour. Randomize headers to look like real people coming from schools, office buildings, etc... Lots of work but possible.

Re: Ask HN: What do people use to prevent crawlers?

#80

Post by alecco »

Rjevski wrote: post #70

Don't prevent them. The same data you let humans access for free should be accessible via bots. If you only want to give out a "reasonable" amount of data, that humans wouldn't usually exceed but bots would, then define a rate-limit that wouldn't inconvenience humans and then apply it for everyone - bot or not. That way you're discriminating based on the amount of data instead of whether it's a bot or not. It will th…

They kill your bandwidth. For a client's catalog site we discovered crawlers were more than half of the used bandwidth costs.
Post reply on HN