A Facebook crawler was making 7M requests per day to my stupid website
coding.napolux.com
A Facebook crawler was making 7M requests per day to my stupid website
1–10 of 416 posts
Re: A Facebook crawler was making 7M requests per day to my stupid website
#2Was it a real FB crawler or a random buggy homemade bot masquerading as one?
Check here: https://developers.facebook.com/docs/sharing/webmasters/craw...
Re: A Facebook crawler was making 7M requests per day to my stupid website
#3Did you check the ip and not just the UA? Does it matches with the one from fb?
Re: A Facebook crawler was making 7M requests per day to my stupid website
#4It would be nice to understand how you came to the conclusion that this was a Facebook bot.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#5This is actually a common tactic from malicious actors - pretend to be other bots to get through to websites. If you reverse lookup the IP address you can see if it part of the facebook network.
UA is very untrustworthy
Re: A Facebook crawler was making 7M requests per day to my stupid website
#6Why does FB need a crawler? Their users provide the content for their site. Is there an FB web search engine?
Re: A Facebook crawler was making 7M requests per day to my stupid website
#7Why does FB need a crawler? Their users provide the content for their site. Is there an FB web search engine?
When you paste in a URL to share it with your friends, Facebook tries to grab some information from that webpage to provide a summary.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#8Unrelated, but has anyone written a Chrome/Firefox extension to browse the web sending out Googlebot or Facebook user agent?
I wonder if you can bypass paywalls or see things that aren't generally presented to regular users
Re: A Facebook crawler was making 7M requests per day to my stupid website
#9Same thing has happened to me: https://twitter.com/moonscript/status/1124888489298808834
The network address range falls under Facebook's ownership, so I don't think it's someone spoofing. I do think it's very possible someone found a way to trigger crawl requests in large quantity. Alternatively, I would not be surprised it's just a bug on facebook's end.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#10Unrelated, but has anyone written a Chrome/Firefox extension to browse the web sending out Googlebot or Facebook user agent? I wonder if you can bypass paywalls or see things that aren't generally presented to regular users
Safari has this built in:
https://osxdaily.com/2013/01/16/change-user-agent-chrome-saf...