Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Facebook crawler was making 7M requests per day to my stupid website

coding.napolux.com

Post reply on HN
Print view View original

81–90 of 416 posts

Re: A Facebook crawler was making 7M requests per day to my stupid website

#81

Post by yupyup54133 »

yupyup54133 wrote: post #79
bberenberg wrote: post #61

Doesn't ignoring headers and taking someones site down fall afoul of the CFAA? Especially given how comments here are showing that this is a recurring issue? Depending on which side of the fence you're on, there could be standing to go after FB for this either for money, or for their lawyers to help set a precedent to limit CFAA further.

When someone does this to Facebook its malicious and they go to jail. WHen Facebook does it to someone else... "oops".

Never forget Aaron Swartz. All he did was send download requests to JSTOR. https://www.youtube.com/watch?v=9vz06QO3UkQ

Re: A Facebook crawler was making 7M requests per day to my stupid website

#84

Post by occamrazor »

yupyup54133 wrote: post #79
bberenberg wrote: post #61

Doesn't ignoring headers and taking someones site down fall afoul of the CFAA? Especially given how comments here are showing that this is a recurring issue? Depending on which side of the fence you're on, there could be standing to go after FB for this either for money, or for their lawyers to help set a precedent to limit CFAA further.

When someone does this to Facebook its malicious and they go to jail. WHen Facebook does it to someone else... "oops".

From the point of view of the law, the intent (malicious or benevolent) is often as important as the action itself.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#85

Post by lclarkmichalek »

Make sure you file a bug - there are a myriad of sources internally, but we can often hunt it down easily enough (assuming it gets triaged to eng). Important info is the host of the urls being crawled, and the User Agent attached to the requests (headers are also good too). A timeseries graph of the hits (with date & timezone specified) can also help.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#86

Post by netsharc »

compiler-guy wrote: post #23
systemvoltage wrote: post #13

Earlier quoted context omitted.

This pattern needs to die. Whenever I paste a link in any sort of messaging app - iMessages to Slack - it puts a thumbnail and summary, polluting the entire conversation with tons of noise. Messaging apps have no business in looking up the URL. Just let it pass as a link. Fuck everything about this and we need to push back on this nonsense.

Part of the goal here is to prevent people from clicking on malicious links. A preview helps with that. Or, more simply, you can't be rickrolled if you know the destination in advance.

The next level would be for the server to see who is requesting content. Facebook IP? "Here's some harmless HTML!". Browser IP? "Here's an executable pretending to be a harmless page!"

Re: A Facebook crawler was making 7M requests per day to my stupid website

#87

Post by occamrazor »

systemvoltage wrote: post #35
crazygringo wrote: post #20

Earlier quoted context omitted.

I disagree completely, I find it extremely useful. Most links are unreadable (e.g. an ID to a cloud file), and are truncated anyways even if readable. The preview gives me the title of the webpage, which is infinitely more useful -- especially letting me know whether it's a document I've already seen (and don't need to open) or something new, and if so, what.

To me it pollutes the discussion away from the messages and puts a bunch of thumbnails where it could just be a clean text conversation. The thing that bothers me is also that I, as a user, have no choice - it just does it automatically. Do you like IRC?

Whatsapp allows to delete the link box: enter the link, wait for the preview, backspace, and send. The hyperlink remains clickable too.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#88

Post by napolux »

lclarkmichalek wrote: post #85

Make sure you file a bug - there are a myriad of sources internally, but we can often hunt it down easily enough (assuming it gets triaged to eng). Important info is the host of the urls being crawled, and the User Agent attached to the requests (headers are also good too). A timeseries graph of the hits (with date & timezone specified) can also help.

Where? Here? https://developers.facebook.com/support/bugs/

Re: A Facebook crawler was making 7M requests per day to my stupid website

#89

Post by cdown »

napolux wrote: post #65
lun4r wrote: post #29

We've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (used "spoofed" UAs) but were all coming from FB owned netblocks. I've contacted FB about it, but they couldn't figure out why this was happening and didn't solve the problem. I found out that there is an option in the FB Catalog manager that lets FB auto-remove…

Thanks man! I'll have a look.

Hey! Facebook engineer here. If you have it, can you send me the User-Agent for these requests? That would definitely help speed up narrowing down what's happening here. If you can provide me the hostname being requested in the Host header, that would be great too.

I just sent you an e-mail, you can also reply to that instead if you prefer not to share those details here. :-)

Re: A Facebook crawler was making 7M requests per day to my stupid website

#90

Post by falcolas »

lclarkmichalek wrote: post #85

Make sure you file a bug - there are a myriad of sources internally, but we can often hunt it down easily enough (assuming it gets triaged to eng). Important info is the host of the urls being crawled, and the User Agent attached to the requests (headers are also good too). A timeseries graph of the hits (with date & timezone specified) can also help.

Why is this the owner's problem? Someone at Facebook should be filing the bug, or better yet instrumenting their systems so that incidents like this issue a wake-up-an-engineer alert.

Fuck this culture of "it's up to the victim of our fuckup to file a bug report with us".

Post reply on HN