Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Facebook crawler was making 7M requests per day to my stupid website

coding.napolux.com

Post reply on HN
Print view View original

271–280 of 416 posts

Re: A Facebook crawler was making 7M requests per day to my stupid website

#271

Post by missedthecue »

wheresmycraisin wrote: post #261
lun4r wrote: post #29

We've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (used "spoofed" UAs) but were all coming from FB owned netblocks. I've contacted FB about it, but they couldn't figure out why this was happening and didn't solve the problem. I found out that there is an option in the FB Catalog manager that lets FB auto-remove…

Side question: how did you get in contact with facebook? I've an ad account that was suspended last year and gave up trying to contact them.

Really? I've never had problems contacting them by email. They're one of the easiest tech companies to talk to.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#272

Post by notriddle »

djxfade wrote: post #207
golem14 wrote: post #198

Earlier quoted context omitted.

I believe, but have not tried, that you can craft zipped files that are small but when expanding produce multi-gigabyte files. If you can sufficiently target the bad guys, you can probably grind them to a halt by serving only them these files. Some care needs to be taken so you don’t harm the innocent and don’t run afoul of any laws that may apply. Good luck.

Do you think it would be possible to create something similar for gzip? If you then serve with Content-Type: text/html, and Content-Encoding: gzip, the client would accept the payload. And when it tries to expand it, it would get expanded to a large file, eating up their resources.

A single layer of deflate compression has a theoretical expansion limit of 1032:1. ZIP bombs with higher ratios only achieve it by nesting a ZIP file inside of another ZIP file and expecting the decompressor to be recursive.

This means you can serve 1M payload and have it come out to 1G at decompression time. Not a bad compression ratio, but it doesn't seem like enough to break Facebook servers without taking on considerable load of your own.

http://www.zlib.net/zlib_tech.html

Re: A Facebook crawler was making 7M requests per day to my stupid website

#273

Post by Ambroos »

laurentdc wrote: post #8

Unrelated, but has anyone written a Chrome/Firefox extension to browse the web sending out Googlebot or Facebook user agent? I wonder if you can bypass paywalls or see things that aren't generally presented to regular users

I surfed through a Google network for a bit a few years ago, and it does reveal things. The most common was Drupal and WordPress sites with some malicious code running that would link to online pharmacies on every page, but only to Google user agents / networks, I assume to boost their ranking. I let a bunch of webmasters know, most of them didn't believe me because they didn't see it themselves.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#274

Post by sammitch »

jrockway wrote: post #134
jokoon wrote: post #108

That's 81 request per second on average. Shouldn't anybody doing such thing be liable, and be sued for negligence and required to pay damages? That sounds like a lot of bandwidth (and server stress).

Shouldn't any public facing website have a rate limit? If someone attempts to circumvent simple rate limits (randomizing the source IP or header content), then that could demonstrate intent to cause damage, and you'd have a better case. But if you don't set a limit, how can you be mad that someone exceeded it? (I know they're ignoring robots.txt, but robots.txt is not a law. And, it doesn't apply to user-generated re…

Plenty of crawlers, including Facebook's, are operating out of huge pools of IPs usually smeared across multiple blocks. If you have an idea on how to ratelimit a crawler doing 2 req/s from 300+ distinct IPs with random spoofed UAs let me know when your startup launches.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#275

Post by xg15 »

stiray wrote: post #216
lun4r wrote: post #29

We've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (used "spoofed" UAs) but were all coming from FB owned netblocks. I've contacted FB about it, but they couldn't figure out why this was happening and didn't solve the problem. I found out that there is an option in the FB Catalog manager that lets FB auto-remove…

I dont really understand what is the issue. On my welcome page (while all other urls are impossible to guess) i give browser something that requires a few seconds of cpu at 100% to crunch. And tracking some user action in between, visting tarpitted urls etc. In last few years no bot came through. Why bother with robots.txt, just give them something to break their teeths... (I would give you the url, but I just dont w…

> but I just dont want ti be visited

Maybe you should just take your website offline?

Re: A Facebook crawler was making 7M requests per day to my stupid website

#276

Post by golem14 »

djxfade wrote: post #207
golem14 wrote: post #198

Earlier quoted context omitted.

I believe, but have not tried, that you can craft zipped files that are small but when expanding produce multi-gigabyte files. If you can sufficiently target the bad guys, you can probably grind them to a halt by serving only them these files. Some care needs to be taken so you don’t harm the innocent and don’t run afoul of any laws that may apply. Good luck.

Do you think it would be possible to create something similar for gzip? If you then serve with Content-Type: text/html, and Content-Encoding: gzip, the client would accept the payload. And when it tries to expand it, it would get expanded to a large file, eating up their resources.

That was the idea. There's a name for it - zip bomb. https://en.wikipedia.org/wiki/Zip_bomb

A superficial search leads to things like https://www.rapid7.com/db/modules/auxiliary/dos/http/gzip_bo...

https://stackoverflow.com/questions/1459673

You really want to be careful about potentially breaking laws ...

Re: A Facebook crawler was making 7M requests per day to my stupid website

#277

Post by bzb3 »

yjftsjthsd-h wrote: post #266
detaro wrote: post #241

Earlier quoted context omitted.

Possibly trying to avoid people sending them a different version of the page than users would see (of course they could change the page after the initial caching of a preview, but Twitter might refresh/check them later). Also, you often need an impressive amount of the stuff thats in a normal UA string for random sites to not break/send you the "unsupported browser, please use Netscape 4 or newer!!!" page/..., althou…

Yeah, I once tried to tell my browser to send... I forget; either no UA, or a blank UA string, or nonsense or just "Firefox" or something. I figured, "hey, some sites might break, but it can't be that important!" It broke everything . IIRC, the breaking point was that my own server refused to talk to me. Now, I still think this is insane, but apparently this really is how it is right now.

You removed a vital part of the http protocol and you're surprised when things break?

Re: A Facebook crawler was making 7M requests per day to my stupid website

#278

Post by redeux »

missedthecue wrote: post #271
wheresmycraisin wrote: post #261

Earlier quoted context omitted.

Side question: how did you get in contact with facebook? I've an ad account that was suspended last year and gave up trying to contact them.

Really? I've never had problems contacting them by email. They're one of the easiest tech companies to talk to.

In my experience they're one of the most useless and difficult companies I've ever tried to interact with.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#279

Post by teknopaul »

jimmaswell wrote: post #193
Drakim wrote: post #185

Earlier quoted context omitted.

Copyleft is about maximizing the users's rights, not the developers's rights. Companies can't take linux, put it on a router for sale, and then say that their customers/users aren't allowed to know what's going on on the box in terms of backdoors and spying. The users have a right to look at the source code if they wish.

Users don't care about any of that, but developers are certainly hindered by restrictive licenses, which in turn hurts users. I never found a right to see source code compelling as a real right. To read the assembly and modify something they bought, sure, but not an entitlement to the source.

Users and developers are not different beasts. Users with source code are more likely to become developers. Developers are more likely to become users if they can tinker.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#280

Post by purple_ferret »

cdown wrote: post #89
napolux wrote: post #65

Earlier quoted context omitted.

Thanks man! I'll have a look.

Hey! Facebook engineer here. If you have it, can you send me the User-Agent for these requests? That would definitely help speed up narrowing down what's happening here. If you can provide me the hostname being requested in the Host header, that would be great too. I just sent you an e-mail, you can also reply to that instead if you prefer not to share those details here. :-)

Please ask Zuck to resign while you're at it. Thanks.
Post reply on HN