Live data from Hacker News

A Facebook crawler was making 7M requests per day to my stupid website

coding.napolux.com

91–100 of 416 posts

Re: A Facebook crawler was making 7M requests per day to my stupid website

#91
post #82

What would happen if you created a "rabbithole" for webcrawlers? Would they fall for it?

Modern web crawlers usually have mechanisms against rabbit holes, e.g. for Apache Nutch you can configure it should only follow to a certain depth of links from a root, it should only follow a redirect a few times and so on. It's always a trade-off though. If you cut off too aggressive you don't get everything, if you cut off too late you waste resources.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#92

It's an interesting thing to think about, that someone can make a request to your website and you pay for it. This is not how the mail works, where someone needs to buy a stamp to spam you.

This is a common pattern in (blocking i/o versus non-blocking i/o) and (sync task versus async task).

The typical analogy I hear for DDoS attacks is if you own a business and a giant group of protesters drives traffic to your store, but no one is interested in buying. They create a backlog to enter the store, they crowd the isles, they pick up your inventory, and the lines and crowds scare off your legitimate visitors/customers.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#93
post #72
post #57

Earlier quoted context omitted.

Doesn't seem the case https://apps.db.ripe.net/db-web-ui/query?searchtext=2a03:288...

IP packets have a source field, which can be fake and not their actual IP. That's a Facebook IP, but the packet might not have actually come from it.

UDP packets can fake their source IP. TCP packets realistically can't.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#94

It's an interesting thing to think about, that someone can make a request to your website and you pay for it. This is not how the mail works, where someone needs to buy a stamp to spam you.

My office still gets spam faxes to this day. The paper and toner only add up to a few cents a month, so it's not worth doing anything about.

I knew a realtor that had a sheet of black paper with a few choice expletives written on it that they would send back to spammers. There was an art to taping it into a loop so it would continuously feed. This was a few decades ago when a the spam faxes could cost more than a stamp.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#95
post #89
post #65

Earlier quoted context omitted.

Thanks man! I'll have a look.

Hey! Facebook engineer here. If you have it, can you send me the User-Agent for these requests? That would definitely help speed up narrowing down what's happening here. If you can provide me the hostname being requested in the Host header, that would be great too. I just sent you an e-mail, you can also reply to that instead if you prefer not to share those details here. :-)

[deleted]

Re: A Facebook crawler was making 7M requests per day to my stupid website

#96
post #69

How do you monetize a robot?

Establish contractual damages in a ToS for the site. Prove violation and offender. Take to court and collect damages.

Converting the effort into cash is tough, but the strategy exists.

Project HoneyPot is an API which allows any website to do this for honeypot email addresses which are injected the website, along with a ToS which says:

> By continuing to access the Website, You acknowledge and agree that each email address the Website contains has a value not less than US $50 derived from their relative secrecy.[1]

[1] https://www.projecthoneypot.org/terms_of_use.php

Re: A Facebook crawler was making 7M requests per day to my stupid website

#99
post #89
post #65

Earlier quoted context omitted.

Thanks man! I'll have a look.

Hey! Facebook engineer here. If you have it, can you send me the User-Agent for these requests? That would definitely help speed up narrowing down what's happening here. If you can provide me the hostname being requested in the Host header, that would be great too. I just sent you an e-mail, you can also reply to that instead if you prefer not to share those details here. :-)

I'm not sure I'd publicly post my email like that, if I worked at FB. But congratulations on your promotion to "official technical contact for all facebook issues forever".

Re: A Facebook crawler was making 7M requests per day to my stupid website

#100
post #89

Earlier quoted context omitted.

Hey! Facebook engineer here. If you have it, can you send me the User-Agent for these requests? That would definitely help speed up narrowing down what's happening here. If you can provide me the hostname being requested in the Host header, that would be great too. I just sent you an e-mail, you can also reply to that instead if you prefer not to share those details here. :-)

I'm not sure I'd publicly post my email like that, if I worked at FB. But congratulations on your promotion to "official technical contact for all facebook issues forever".

My e-mail address is already public from my kernel commits and upstream work. :-)
Post reply on HN