Live data from Hacker News

A Facebook crawler was making 7M requests per day to my stupid website

coding.napolux.com

411–416 of 416 posts

Re: A Facebook crawler was making 7M requests per day to my stupid website

#411
post #168

Earlier quoted context omitted.

I just had an idea: if you control your own name server I believe you could use a BIND view to send all their own traffic to themselves based on the source address. By the way, if someone discovers how to trigger this issue it would be easy to use it as a DOS pseudo-botnet.

You could just probably send an HTTP redirect to their own site, no need to play with DNS for that

Probably better to find the highest level FB employees personal site that you can and send the million requests from fb there.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#412

Earlier quoted context omitted.

robots.txt is not a legal document. It is asking nicely, and plenty of crawlers purposefully ignore it.

Legal document is a tricky phrase to use. "No trespassing" signs are usually considered sufficient to justify prosecution for trespassing. If the sign is conspicuously placed, it does not usually matter if you actually see the sign or not. I am not as familiar with law around accessing computer systems, but I imagine that given some of the draconian enforcement we've seen in the past that a robots.txt should be suffi…

the no tresspassing sign does not mean i can't yell at you from the street 'tell me your life story.' which you are free to ignore.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#413
post #327

Earlier quoted context omitted.

FB uses Office365 for email. It was on-premise Exchange many many years ago, but moved "to the cloud" a while back.

Feels odd to read Facebook uses office365/exchange for emails. they haven't built their fsuite yet, I thought they would simply promote Facebook messenger internally. I'm only half joking.

Most communication is via Workplace (group posts and chat). Emails aren't very common any more - mainly for communication with external people and alerting.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#414
post #306

I’m a bit confused. Facebook is social media company, why do they send out crawlers?

Facebook shows previews of pretty much everything you share a link to, so at a minimum they have to fetch the page/image to build those previews.

I see, thanks for the explanation.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#415

Earlier quoted context omitted.

Legal document is a tricky phrase to use. "No trespassing" signs are usually considered sufficient to justify prosecution for trespassing. If the sign is conspicuously placed, it does not usually matter if you actually see the sign or not. I am not as familiar with law around accessing computer systems, but I imagine that given some of the draconian enforcement we've seen in the past that a robots.txt should be suffi…

the no tresspassing sign does not mean i can't yell at you from the street 'tell me your life story.' which you are free to ignore.

Sure, but federal law does prevent you from repeatedly placing that request into my mailbox. I don't even need a sign for that.

Making an http request does not fit cleanly as an analogue to yelling from the street, nor does it fit as an analogue to throwing a written request on a brick through a window. It is something different that must be understood on its own terms.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#416
post #375

Earlier quoted context omitted.

I had a different idea. Maybe you could craft a zip-bomb response. The bot would fetch the small gzipped content and upon extraction discover it was GBs of data? Not sure that's possible here, when responding to a request, but that would surely turn the admins attention to it.

Here's an example of things you can do against malicious crawlers: http://www.hackerfactor.com/blog/index.php?/archives/762-Att... .

Thanks a lot! I've spent like 2h now reading his blog. Just amazing.
Post reply on HN