Earlier quoted context omitted.
I just had an idea: if you control your own name server I believe you could use a BIND view to send all their own traffic to themselves based on the source address. By the way, if someone discovers how to trigger this issue it would be easy to use it as a DOS pseudo-botnet.
You could just probably send an HTTP redirect to their own site, no need to play with DNS for that
A Facebook crawler was making 7M requests per day to my stupid website
411–416 of 416 posts
Re: A Facebook crawler was making 7M requests per day to my stupid website
#412Earlier quoted context omitted.
robots.txt is not a legal document. It is asking nicely, and plenty of crawlers purposefully ignore it.
Legal document is a tricky phrase to use. "No trespassing" signs are usually considered sufficient to justify prosecution for trespassing. If the sign is conspicuously placed, it does not usually matter if you actually see the sign or not. I am not as familiar with law around accessing computer systems, but I imagine that given some of the draconian enforcement we've seen in the past that a robots.txt should be suffi…
Re: A Facebook crawler was making 7M requests per day to my stupid website
#413Earlier quoted context omitted.
FB uses Office365 for email. It was on-premise Exchange many many years ago, but moved "to the cloud" a while back.
Feels odd to read Facebook uses office365/exchange for emails. they haven't built their fsuite yet, I thought they would simply promote Facebook messenger internally. I'm only half joking.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#414Re: A Facebook crawler was making 7M requests per day to my stupid website
#415Earlier quoted context omitted.
Legal document is a tricky phrase to use. "No trespassing" signs are usually considered sufficient to justify prosecution for trespassing. If the sign is conspicuously placed, it does not usually matter if you actually see the sign or not. I am not as familiar with law around accessing computer systems, but I imagine that given some of the draconian enforcement we've seen in the past that a robots.txt should be suffi…
the no tresspassing sign does not mean i can't yell at you from the street 'tell me your life story.' which you are free to ignore.
Making an http request does not fit cleanly as an analogue to yelling from the street, nor does it fit as an analogue to throwing a written request on a brick through a window. It is something different that must be understood on its own terms.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#416Earlier quoted context omitted.
I had a different idea. Maybe you could craft a zip-bomb response. The bot would fetch the small gzipped content and upon extraction discover it was GBs of data? Not sure that's possible here, when responding to a request, but that would surely turn the admins attention to it.
Here's an example of things you can do against malicious crawlers: http://www.hackerfactor.com/blog/index.php?/archives/762-Att... .