I think sending back a `429 Too Many Requests` can solve this. Unless you want to block their IPs completely, which I doubt you'd want to do.
A Facebook crawler was making 7M requests per day to my stupid website
121–130 of 416 posts
Re: A Facebook crawler was making 7M requests per day to my stupid website
#122Earlier quoted context omitted.
I can add I work in SEO (tech side), so the website is "super optimized", but it's not rocket science.
Super optimized? The site takes longer to load than Hacker News for me.
They explicitly didn't mention the site this affected. It's not the one linked in the OP.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#123Earlier quoted context omitted.
Hey! Facebook engineer here. If you have it, can you send me the User-Agent for these requests? That would definitely help speed up narrowing down what's happening here. If you can provide me the hostname being requested in the Host header, that would be great too. I just sent you an e-mail, you can also reply to that instead if you prefer not to share those details here. :-)
I don't want to share my website for personal reasons, but here is some data from cloudflare dashboard (a request made on 11 Jun, 2020 21:30:55 from Ireland, I have 3 requests in the same second from 2 different IPs) user-agent: facebookexternalhit/1.1 (+ http://www.facebook.com/externalhit_uatext.php ) ip 1: 2a03:2880:22ff:3::face:b00c (1 request) ip 2: 2a03:2880:22ff:b::face:b00c (2 requests) ASN: AS32934 FACEBOOK
Yes requests are still coming. Thanks CloudFlare for saving my ass.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#124Did anyone notice the branded IP address? 2a03:2880:20ff:d::face:b00c "face:b00c"
My "404-pests" fail2ban-client status, which drops everything making *.php requests (This machine has never had PHP installed...):
2a03:2880:10ff:14::face:b00c 2a03:2880:10ff:21::face:b00c 2a03:2880:11ff:1a::face:b00c 2a03:2880:11ff:1f::face:b00c 2a03:2880:11ff:2::face:b00c 2a03:2880:12ff:10::face:b00c 2a03:2880:12ff:1::face:b00c 2a03:2880:12ff:9::face:b00c 2a03:2880:12ff:d::face:b00c 2a03:2880:13ff:3::face:b00c 2a03:2880:13ff:4::face:b00c 2a03:2880:20ff:12::face:b00c 2a03:2880:20ff:1e::face:b00c 2a03:2880:20ff:4::face:b00c 2a03:2880:20ff:5::face:b00c 2a03:2880:20ff:75::face:b00c 2a03:2880:20ff:77::face:b00c 2a03:2880:20ff:e::face:b00c 2a03:2880:21ff:30::face:b00c 2a03:2880:22ff:11::face:b00c 2a03:2880:22ff:12::face:b00c 2a03:2880:22ff:14::face:b00c 2a03:2880:23ff:5::face:b00c 2a03:2880:23ff:b::face:b00c 2a03:2880:23ff:c::face:b00c 2a03:2880:30ff:10::face:b00c 2a03:2880:30ff:11::face:b00c 2a03:2880:30ff:17::face:b00c 2a03:2880:30ff:1::face:b00c 2a03:2880:30ff:71::face:b00c 2a03:2880:30ff:a::face:b00c 2a03:2880:30ff:b::face:b00c 2a03:2880:30ff:c::face:b00c 2a03:2880:30ff:d::face:b00c 2a03:2880:30ff:f::face:b00c 2a03:2880:31ff:10::face:b00c 2a03:2880:31ff:11::face:b00c 2a03:2880:31ff:12::face:b00c 2a03:2880:31ff:13::face:b00c 2a03:2880:31ff:17::face:b00c 2a03:2880:31ff:1::face:b00c 2a03:2880:31ff:2::face:b00c 2a03:2880:31ff:3::face:b00c 2a03:2880:31ff:4::face:b00c 2a03:2880:31ff:5::face:b00c 2a03:2880:31ff:6::face:b00c 2a03:2880:31ff:71::face:b00c 2a03:2880:31ff:7::face:b00c 2a03:2880:31ff:8::face:b00c 2a03:2880:31ff:c::face:b00c 2a03:2880:31ff:d::face:b00c 2a03:2880:31ff:e::face:b00c 2a03:2880:31ff:f::face:b00c 2a03:2880:32ff:4::face:b00c 2a03:2880:32ff:5::face:b00c 2a03:2880:32ff:70::face:b00c 2a03:2880:32ff:d::face:b00c 2a03:2880:ff:16::face:b00c 2a03:2880:ff:17::face:b00c 2a03:2880:ff:1a::face:b00c 2a03:2880:ff:1c::face:b00c 2a03:2880:ff:1d::face:b00c 2a03:2880:ff:25::face:b00c 2a03:2880:ff::face:b00c 2a03:2880:ff:b::face:b00c 2a03:2880:ff:c::face:b00c 2a03:2880:ff:d::face:b00c
Re: A Facebook crawler was making 7M requests per day to my stupid website
#125Earlier quoted context omitted.
When someone does this to Facebook its malicious and they go to jail. WHen Facebook does it to someone else... "oops".
From the point of view of the law, the intent (malicious or benevolent) is often as important as the action itself.
The always-classic "what color are your bits?"
Re: A Facebook crawler was making 7M requests per day to my stupid website
#126Earlier quoted context omitted.
They've done this before to me, too. First I tried `iptables -j DROP`, which made the machine somewhat usable, but didn't help with the traffic. After trying a few things, I tried `-j TARPIT`, and that appeared to make them back off. Of course, sample size of 1, etc. It could have been coincidental.
Tarpits are an underappreciated solution to a pool of bad actors. You can add artificial wait times to responses, or you can just route all of the 'bad' traffic to one machine, which becomes oversubscribed (be sure to segregate your stats!). All bad actors fighting over the same scraps creates proportional backpressure. Just adding 2 second delays to each request won't necessarily achieve that if multiple user agents…
I never looked into the TARPIT option in iptables before reading your comment. That seems really useful. I've been dealing with on and off bursts of traffic from a single AWS region for the last month. They usually keep going for about 90 minutes every day, regardless of how many IPs I block, and consume every available resource with about 250 requests per second (not a big server and I'm still waiting for approval to just outright block the AWS region). I'm going to try a tarpit next time rather than a DROP and see if it makes a difference.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#127Earlier quoted context omitted.
Super optimized? The site takes longer to load than Hacker News for me.
They explicitly didn't mention the site this affected. It's not the one linked in the OP.
Super optimized with Facebook button on each post then. Not my definition of 'super optimized', but you know.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#128How do you monetize a robot?
Establish contractual damages in a ToS for the site. Prove violation and offender. Take to court and collect damages. Converting the effort into cash is tough, but the strategy exists. Project HoneyPot is an API which allows any website to do this for honeypot email addresses which are injected the website, along with a ToS which says: > By continuing to access the Website, You acknowledge and agree that each email a…
Has that ever worked? I can't find any record of judgments one way or the other, on their website or elsewhere.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#129Earlier quoted context omitted.
They explicitly didn't mention the site this affected. It's not the one linked in the OP.
Super optimized with Facebook button on each post then. Not my definition of 'super optimized', but you know.
Super optimized in a SEO sense, not in a pleasing the HN crowd kind of sense I'd assume ;)
Re: A Facebook crawler was making 7M requests per day to my stupid website
#130I think sending back a `429 Too Many Requests` can solve this. Unless you want to block their IPs completely, which I doubt you'd want to do.
It says in the post he did this already