What would happen if you created a "rabbithole" for webcrawlers? Would they fall for it?
A Facebook crawler was making 7M requests per day to my stupid website
91–100 of 416 posts
Re: A Facebook crawler was making 7M requests per day to my stupid website
#92It's an interesting thing to think about, that someone can make a request to your website and you pay for it. This is not how the mail works, where someone needs to buy a stamp to spam you.
The typical analogy I hear for DDoS attacks is if you own a business and a giant group of protesters drives traffic to your store, but no one is interested in buying. They create a backlog to enter the store, they crowd the isles, they pick up your inventory, and the lines and crowds scare off your legitimate visitors/customers.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#93Earlier quoted context omitted.
Doesn't seem the case https://apps.db.ripe.net/db-web-ui/query?searchtext=2a03:288...
IP packets have a source field, which can be fake and not their actual IP. That's a Facebook IP, but the packet might not have actually come from it.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#94It's an interesting thing to think about, that someone can make a request to your website and you pay for it. This is not how the mail works, where someone needs to buy a stamp to spam you.
I knew a realtor that had a sheet of black paper with a few choice expletives written on it that they would send back to spammers. There was an art to taping it into a loop so it would continuously feed. This was a few decades ago when a the spam faxes could cost more than a stamp.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#95Earlier quoted context omitted.
Thanks man! I'll have a look.
Hey! Facebook engineer here. If you have it, can you send me the User-Agent for these requests? That would definitely help speed up narrowing down what's happening here. If you can provide me the hostname being requested in the Host header, that would be great too. I just sent you an e-mail, you can also reply to that instead if you prefer not to share those details here. :-)
Re: A Facebook crawler was making 7M requests per day to my stupid website
#96How do you monetize a robot?
Converting the effort into cash is tough, but the strategy exists.
Project HoneyPot is an API which allows any website to do this for honeypot email addresses which are injected the website, along with a ToS which says:
> By continuing to access the Website, You acknowledge and agree that each email address the Website contains has a value not less than US $50 derived from their relative secrecy.[1]
Re: A Facebook crawler was making 7M requests per day to my stupid website
#97Re: A Facebook crawler was making 7M requests per day to my stupid website
#98isn't this what robots.txt for? can't you block facebook?
Re: A Facebook crawler was making 7M requests per day to my stupid website
#99Earlier quoted context omitted.
Thanks man! I'll have a look.
Hey! Facebook engineer here. If you have it, can you send me the User-Agent for these requests? That would definitely help speed up narrowing down what's happening here. If you can provide me the hostname being requested in the Host header, that would be great too. I just sent you an e-mail, you can also reply to that instead if you prefer not to share those details here. :-)
Re: A Facebook crawler was making 7M requests per day to my stupid website
#100Earlier quoted context omitted.
Hey! Facebook engineer here. If you have it, can you send me the User-Agent for these requests? That would definitely help speed up narrowing down what's happening here. If you can provide me the hostname being requested in the Host header, that would be great too. I just sent you an e-mail, you can also reply to that instead if you prefer not to share those details here. :-)
I'm not sure I'd publicly post my email like that, if I worked at FB. But congratulations on your promotion to "official technical contact for all facebook issues forever".