This reminds me of a case years ago when I had a small-ish video file on my server that would cause Google Bot to just keep fetching it in a loop. It wasted a huge amount of bandwidth for me since I only noticed it in a report at the end of the month. I had no way of figuring out what was wrong with it, so I just deleted the file and the bot went away.
A Facebook crawler was making 7M requests per day to my stupid website
11–20 of 416 posts
Re: A Facebook crawler was making 7M requests per day to my stupid website
#12Why does FB need a crawler? Their users provide the content for their site. Is there an FB web search engine?
When you paste in a URL to share it with your friends, Facebook tries to grab some information from that webpage to provide a summary.
What happens if that link performs an action upon a GET request?
Edit: Folks, I agree with you all, but I've seen a lot of garbage out there. Just asking the question for the discussion.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#13Why does FB need a crawler? Their users provide the content for their site. Is there an FB web search engine?
When you paste in a URL to share it with your friends, Facebook tries to grab some information from that webpage to provide a summary.
This pattern needs to die.
Whenever I paste a link in any sort of messaging app - iMessages to Slack - it puts a thumbnail and summary, polluting the entire conversation with tons of noise.
Messaging apps have no business in looking up the URL. Just let it pass as a link.
Fuck everything about this and we need to push back on this nonsense.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#14Earlier quoted context omitted.
When you paste in a URL to share it with your friends, Facebook tries to grab some information from that webpage to provide a summary.
What happens if that link performs an action upon a GET request? Edit: Folks, I agree with you all, but I've seen a lot of garbage out there. Just asking the question for the discussion.
That "link" doesn't respect the GET semantics then.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#15Earlier quoted context omitted.
When you paste in a URL to share it with your friends, Facebook tries to grab some information from that webpage to provide a summary.
What happens if that link performs an action upon a GET request? Edit: Folks, I agree with you all, but I've seen a lot of garbage out there. Just asking the question for the discussion.
that's your own problem because you are going against the HTTP protocol standard. GET should be idempotent.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#16Earlier quoted context omitted.
When you paste in a URL to share it with your friends, Facebook tries to grab some information from that webpage to provide a summary.
What happens if that link performs an action upon a GET request? Edit: Folks, I agree with you all, but I've seen a lot of garbage out there. Just asking the question for the discussion.
Then like literally every search engine and website crawler ever does, the action will be performed.
Which is why it's bad practice to design your website using GET for actions. That's what POST is for.
I mean, using GET for actions will break so many things -- browser prefetching, link previews, the list is endless. If you use GET for actions, just... yikes.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#17Earlier quoted context omitted.
When you paste in a URL to share it with your friends, Facebook tries to grab some information from that webpage to provide a summary.
What happens if that link performs an action upon a GET request? Edit: Folks, I agree with you all, but I've seen a lot of garbage out there. Just asking the question for the discussion.
Then it performs the action, what else would it do?
Re: A Facebook crawler was making 7M requests per day to my stupid website
#18Earlier quoted context omitted.
When you paste in a URL to share it with your friends, Facebook tries to grab some information from that webpage to provide a summary.
This pattern needs to die. Whenever I paste a link in any sort of messaging app - iMessages to Slack - it puts a thumbnail and summary, polluting the entire conversation with tons of noise. Messaging apps have no business in looking up the URL. Just let it pass as a link. Fuck everything about this and we need to push back on this nonsense.
I like link previews. I don't like when they are managed server-side instead of client-side.
Re: A Facebook crawler was making 7M requests per day to my stupid website
#19Earlier quoted context omitted.
When you paste in a URL to share it with your friends, Facebook tries to grab some information from that webpage to provide a summary.
What happens if that link performs an action upon a GET request? Edit: Folks, I agree with you all, but I've seen a lot of garbage out there. Just asking the question for the discussion.
Then the action needs to be harmless, because GET is defined as not merely idempotent but also safe.
Didn't we already learn this lesson after all the unsafe-GET problems unveiled when prefetching browser accelerators came on the scene in, IIRC, the late 1990s?
Re: A Facebook crawler was making 7M requests per day to my stupid website
#20Earlier quoted context omitted.
When you paste in a URL to share it with your friends, Facebook tries to grab some information from that webpage to provide a summary.
This pattern needs to die. Whenever I paste a link in any sort of messaging app - iMessages to Slack - it puts a thumbnail and summary, polluting the entire conversation with tons of noise. Messaging apps have no business in looking up the URL. Just let it pass as a link. Fuck everything about this and we need to push back on this nonsense.
I disagree completely, I find it extremely useful.
Most links are unreadable (e.g. an ID to a cloud file), and are truncated anyways even if readable.
The preview gives me the title of the webpage, which is infinitely more useful -- especially letting me know whether it's a document I've already seen (and don't need to open) or something new, and if so, what.