Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Facebook crawler was making 7M requests per day to my stupid website

coding.napolux.com

Post reply on HN
Print view View original

291–300 of 416 posts

Re: A Facebook crawler was making 7M requests per day to my stupid website

#291

Post by zucker42 »

bzb3 wrote: post #277
yjftsjthsd-h wrote: post #266

Earlier quoted context omitted.

Yeah, I once tried to tell my browser to send... I forget; either no UA, or a blank UA string, or nonsense or just "Firefox" or something. I figured, "hey, some sites might break, but it can't be that important!" It broke everything . IIRC, the breaking point was that my own server refused to talk to me. Now, I still think this is insane, but apparently this really is how it is right now.

You removed a vital part of the http protocol and you're surprised when things break?

Seems like it follows the spec to me:

https://tools.ietf.org/html/rfc7231#section-5.5.3

Re: A Facebook crawler was making 7M requests per day to my stupid website

#294

Post by x0 »

golem14 wrote: post #198
napolux wrote: post #132

Earlier quoted context omitted.

I think we're together in this, my friend

I believe, but have not tried, that you can craft zipped files that are small but when expanding produce multi-gigabyte files. If you can sufficiently target the bad guys, you can probably grind them to a halt by serving only them these files. Some care needs to be taken so you don’t harm the innocent and don’t run afoul of any laws that may apply. Good luck.

I believe zip bombs are pretty easily mitigated, with memory limits, cpu time ulimits, etc.

But zip bombs aren't limited to zip files... lots of files have some compression in them You can make malicious PNG's that do the same thing. Probably tonnes of other files.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#295

Post by itsjloh »

When I enabled IPv6 traffic at home and logged default denies I constantly see Facebook owned IPv6 blocks trying to reach addresses browsing Facebook.

I have no idea what its trying to do but its legitimately the only IPv6 inbound traffic that isn't related to normal browsing.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#296

Post by x0 »

mcguire wrote: post #70
alpb wrote: post #63

Hate to say but this is probably a FB engineer running a test/experiment and not a production crawler taking robots.txt etc into account.

Or malware infecting some FB-internal machine?

Doubt it. If you're a bad enough dude to get control over Facebook's internal infrastructure, I doubt you'd blow that access by using it to spam random sites.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#297

Post by asveikau »

apolymath wrote: post #230
ruuda wrote: post #167

Earlier quoted context omitted.

Thanks, I’m flattered to see it be used as inspiration :) I searched quickly but I didn’t find that repository. You might want to link it somewhere in your footer or from a comment in the html.

Flattered that your few, boring lines of CSS and HTML (that could easily be reproduced by a monkey) was used on someone else's website?

Sounds like yes, they are flattered for exactly that. And it's totally ok.

There exists small joys in life, like being flattered for something that other people might not think are much of a big deal.

I guess that's one reason why something may be called flattering in the first place, if they didn't see it as being all that much in their own eyes, but somebody else appreciates it.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#298

Post by fomine3 »

_lqaf wrote: post #33
leafo wrote: post #9

Same thing has happened to me: https://twitter.com/moonscript/status/1124888489298808834 The network address range falls under Facebook's ownership, so I don't think it's someone spoofing. I do think it's very possible someone found a way to trigger crawl requests in large quantity. Alternatively, I would not be surprised it's just a bug on facebook's end.

They've done this before to me, too. First I tried `iptables -j DROP`, which made the machine somewhat usable, but didn't help with the traffic. After trying a few things, I tried `-j TARPIT`, and that appeared to make them back off. Of course, sample size of 1, etc. It could have been coincidental.

TIL TARIPT!

Re: A Facebook crawler was making 7M requests per day to my stupid website

#299

Post by justusw »

wheresmycraisin wrote: post #261
lun4r wrote: post #29

We've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (used "spoofed" UAs) but were all coming from FB owned netblocks. I've contacted FB about it, but they couldn't figure out why this was happening and didn't solve the problem. I found out that there is an option in the FB Catalog manager that lets FB auto-remove…

Side question: how did you get in contact with facebook? I've an ad account that was suspended last year and gave up trying to contact them.

Similar experience, closed ad account because of “suspicious activity”, at least ten support tickets (half closed automatically), four lame apologies (our system says no, sry) and then finally, “there was an error in our system, you’re good to go”

Re: A Facebook crawler was making 7M requests per day to my stupid website

#300

Post by njvsifhgnvion »

djxfade wrote: post #207
golem14 wrote: post #198

Earlier quoted context omitted.

I believe, but have not tried, that you can craft zipped files that are small but when expanding produce multi-gigabyte files. If you can sufficiently target the bad guys, you can probably grind them to a halt by serving only them these files. Some care needs to be taken so you don’t harm the innocent and don’t run afoul of any laws that may apply. Good luck.

Do you think it would be possible to create something similar for gzip? If you then serve with Content-Type: text/html, and Content-Encoding: gzip, the client would accept the payload. And when it tries to expand it, it would get expanded to a large file, eating up their resources.

Just a guessing, but maybe a really large file with a limited character set, maybe even just repeating 1 character should compress really well
Post reply on HN