Did anyone notice the branded IP address? 2a03:2880:20ff:d::face:b00c
"face:b00c"
A Facebook crawler was making 7M requests per day to my stupid website
111–120 of 416 posts
Re: A Facebook crawler was making 7M requests per day to my stupid website
#112Earlier quoted context omitted.
Thanks man! I'll have a look.
Hey! Facebook engineer here. If you have it, can you send me the User-Agent for these requests? That would definitely help speed up narrowing down what's happening here. If you can provide me the hostname being requested in the Host header, that would be great too. I just sent you an e-mail, you can also reply to that instead if you prefer not to share those details here. :-)
I don't want to share my website for personal reasons, but here is some data from cloudflare dashboard (a request made on 11 Jun, 2020 21:30:55 from Ireland, I have 3 requests in the same second from 2 different IPs)
user-agent: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) ip 1: 2a03:2880:22ff:3::face:b00c (1 request) ip 2: 2a03:2880:22ff:b::face:b00c (2 requests) ASN: AS32934 FACEBOOK
Re: A Facebook crawler was making 7M requests per day to my stupid website
#113Did anyone notice the branded IP address? 2a03:2880:20ff:d::face:b00c "face:b00c"
if only they could have convinced IANA to give them a vanity prefix.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#114Did anyone notice the branded IP address? 2a03:2880:20ff:d::face:b00c "face:b00c"
The mighty power of IPV6 :P
Re: A Facebook crawler was making 7M requests per day to my stupid website
#115Earlier quoted context omitted.
> he was doing it with the intent to redistribute the content for free There is actually no proof of this.
Really? I felt that what happened to Aaron was a tragic injustice on many levels (from the fact that he was charged at all to the number of charges they threw at him), but for some reason I had always thought that it was fairly well-known that that was his intent. I have no idea where I got that impression from, but I do recall reading several articles about him, as well as his blog around that time. It's possible I…
IIRC, he had published a manifesto about how he believed that information that was paid for with public funds should be free for the public to view. However, I don't believe that he ever mentioned why he was making get requests to JSTOR's servers and I know that he never uploaded any of those JSTOR documents to the internet for public download.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#116Earlier quoted context omitted.
> he was doing it with the intent to redistribute the content for free There is actually no proof of this.
Really? I felt that what happened to Aaron was a tragic injustice on many levels (from the fact that he was charged at all to the number of charges they threw at him), but for some reason I had always thought that it was fairly well-known that that was his intent. I have no idea where I got that impression from, but I do recall reading several articles about him, as well as his blog around that time. It's possible I…
Criminal law is about proving things, not in-charachter assumptions.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#117That's 81 request per second on average. Shouldn't anybody doing such thing be liable, and be sued for negligence and required to pay damages? That sounds like a lot of bandwidth (and server stress).
I saw also of 300reqs per second.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#118Same thing has happened to me: https://twitter.com/moonscript/status/1124888489298808834 The network address range falls under Facebook's ownership, so I don't think it's someone spoofing. I do think it's very possible someone found a way to trigger crawl requests in large quantity. Alternatively, I would not be surprised it's just a bug on facebook's end.
This was happening to us > 5 years ago. The FB crawlers were taking out our image serving system as we used the og:image thing. What we did was route FB crawler traffic to a separate Auto Scaling Group to keep our users happy while also getting the nice preview image on FB when our content was shared. I can't understate the volume of the FB requests, I can't remember the exact numbers now but it was insane.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#119Re: A Facebook crawler was making 7M requests per day to my stupid website
#120That's 81 request per second on average. Shouldn't anybody doing such thing be liable, and be sued for negligence and required to pay damages? That sounds like a lot of bandwidth (and server stress).
Yes, this was my first thought too - Facebook is essentially denial of servicing people's sites and it sounds like they're aware of it. Move fast and break your things. If this were your average Joe I'm fairly certain something like the DMCA could be used.