Unrelated, but has anyone written a Chrome/Firefox extension to browse the web sending out Googlebot or Facebook user agent? I wonder if you can bypass paywalls or see things that aren't generally presented to regular users
A Facebook crawler was making 7M requests per day to my stupid website
21–30 of 416 posts
Re: A Facebook crawler was making 7M requests per day to my stupid website
#22Earlier quoted context omitted.
When you paste in a URL to share it with your friends, Facebook tries to grab some information from that webpage to provide a summary.
What happens if that link performs an action upon a GET request? Edit: Folks, I agree with you all, but I've seen a lot of garbage out there. Just asking the question for the discussion.
I'm not sure what sort of action you mean, but Facebook doesn't fetch the page in real-time using the client's browser. Some sort of job scrapes the page HTML looking for metadata and then uses that metadata to populate the preview. If you want to play around with what it "sees", you can test using the sharing debugger: https://developers.facebook.com/tools/debug/
Re: A Facebook crawler was making 7M requests per day to my stupid website
#23Earlier quoted context omitted.
When you paste in a URL to share it with your friends, Facebook tries to grab some information from that webpage to provide a summary.
This pattern needs to die. Whenever I paste a link in any sort of messaging app - iMessages to Slack - it puts a thumbnail and summary, polluting the entire conversation with tons of noise. Messaging apps have no business in looking up the URL. Just let it pass as a link. Fuck everything about this and we need to push back on this nonsense.
Part of the goal here is to prevent people from clicking on malicious links. A preview helps with that.
Or, more simply, you can't be rickrolled if you know the destination in advance.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#24Earlier quoted context omitted.
What happens if that link performs an action upon a GET request? Edit: Folks, I agree with you all, but I've seen a lot of garbage out there. Just asking the question for the discussion.
that's your own problem because you are going against the HTTP protocol standard. GET should be idempotent.
Going against the HTTP standard isn't a problem. For example, it's a good practice to ignore HEAD requests as opposed to responding appropriately.
The problem with unsafe GET is that it conflicts with reality, not that it conflicts with the standard.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#25It would be nice to understand how you came to the conclusion that this was a Facebook bot.
You can check list of IP addresses?
Re: A Facebook crawler was making 7M requests per day to my stupid website
#26Earlier quoted context omitted.
This pattern needs to die. Whenever I paste a link in any sort of messaging app - iMessages to Slack - it puts a thumbnail and summary, polluting the entire conversation with tons of noise. Messaging apps have no business in looking up the URL. Just let it pass as a link. Fuck everything about this and we need to push back on this nonsense.
I like link previews. I don't like when they are managed server-side instead of client-side.
Caching a link's title and thumbnail server-side will save your site potentially millions of requests from Facebook and elsewhere.
Seems like a benefit for site owners to me.
Not to mention that the server is reducing the image size of a thumbnail, potentially converting from HTTP to HTTPS, and so on.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#27If you can verify it's actually coming from Facebook's IP address (as many other comments here suggest), you should absolutely get in touch with them, though I'm not sure how. Perhaps their security team. That's a very serious bug that they'd be glad to catch -- if it's affecting you it's surely affecting others.
Otherwise it's a bot/malware/etc. spoofing Facebook and gone wrong, which sucks. And yeah just block it by UA, and hopefully eventually it goes away.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#28Earlier quoted context omitted.
What happens if that link performs an action upon a GET request? Edit: Folks, I agree with you all, but I've seen a lot of garbage out there. Just asking the question for the discussion.
Then the action needs to be harmless, because GET is defined as not merely idempotent but also safe. Didn't we already learn this lesson after all the unsafe-GET problems unveiled when prefetching browser accelerators came on the scene in, IIRC, the late 1990s?
> Then the action needs to be harmless to repeat, because GET is defined as idempotent.
No, this is a terrible response. The action needs to be harmless to execute every time, not just every time after the first time.
HTTP DELETE is conceptually idempotent, but you don't want to be deleting stuff with GET requests. That's why the standard provides a DELETE method! The distinction that really matters is safe/unsafe, not idempotent/unique.
(Do you need to use DELETE for deleting stuff? No, POST is fine.)
Re: A Facebook crawler was making 7M requests per day to my stupid website
#29We've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (used "spoofed" UAs) but were all coming from FB owned netblocks.
I've contacted FB about it, but they couldn't figure out why this was happening and didn't solve the problem.
I found out that there is an option in the FB Catalog manager that lets FB auto-remove items from the catalog when their destination page is gone. Disabling this option solved the issue.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#30It would be nice to understand how you came to the conclusion that this was a Facebook bot.
You can check list of IP addresses?
the OP did not (explicitly) check, but you can check if the IP falls into a range allocated to Facebook e.g. https://ipinfo.io/AS32934