Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Facebook crawler was making 7M requests per day to my stupid website

coding.napolux.com

Post reply on HN
Print view View original

361–370 of 416 posts

Re: A Facebook crawler was making 7M requests per day to my stupid website

#361

Post by raverbashing »

pera wrote: post #168
lun4r wrote: post #29

We've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (used "spoofed" UAs) but were all coming from FB owned netblocks. I've contacted FB about it, but they couldn't figure out why this was happening and didn't solve the problem. I found out that there is an option in the FB Catalog manager that lets FB auto-remove…

I just had an idea: if you control your own name server I believe you could use a BIND view to send all their own traffic to themselves based on the source address. By the way, if someone discovers how to trigger this issue it would be easy to use it as a DOS pseudo-botnet.

You could just probably send an HTTP redirect to their own site, no need to play with DNS for that

Re: A Facebook crawler was making 7M requests per day to my stupid website

#362

Post by cjslep »

lovecg wrote: post #347
cjslep wrote: post #196

Earlier quoted context omitted.

It may focus on user's rights, but it still requires technical expertise to exercise half of the 4 freedoms: #1 (inspection & modification) and #3 (distributing your modifications). Merely knowing to ask "can I see the source code" I would put into the "technical user" realm. Overwhelmingly, most people don't know or don't care. The other two freedoms #0 (freedom to run) and #2 (freedom to share) are readily obvious…

The users don't have to be technical to benefit from those freedoms though - there's a level of indirection involved. I might not have personally scrutinized every line of the Linux kernel, but knowing that there are tons of people in the world with the ability and motivation to do that inspires confidence.

I am not arguing that one has to be technical in order to benefit from those freedoms. I like FLOSS and agree users in general benefit.

I am saying that your example, for instance, stills falls under the "how does a non-technical user simply verify the software they just downloaded respects their freedoms?" which is an very real educational and cultural problem. For example, see the massive money and numerous gun ranges, gun stores, gun clubs, and other gun-associated organizations in the US that work to educate the "unskilled" general public on "how to be aware, recognize, and exercise their rights and freedoms" under the 2nd Amendment while respecting local laws. Folks generally are 1) aware they have the right and 2) have a low-friction no-special-technical-skilled path to exercising that right. The FLOSS movement is nowhere near that level of educating and making aware non-technical users of their freedoms, their digital rights, and how to then act upon them and exercise them. Folks generally are 1) unaware of libre software and 2) don't have a low-friction no-special-technical-skilled path to exercising that right.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#363

Post by lun4r »

rob-olmos wrote: post #330
smallgovt wrote: post #188

Earlier quoted context omitted.

>> The bots didn't identify as FB (used "spoofed" UAs) That's surprising. What were the spoofed user agents that they used? We've run into this issue also, but all Facebook bot activity had user agents that contained the string "facebookexternalhit".

I've seen these these two user-agents from FB IPs, maybe others: Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53 Mozilla/5.0 (iPhone; CPU iPhone OS 13_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Mobile/15E148 Safari/604.1 Which also execute javascript in some modified sandbox or something, causing er…

Yep, these were the UAs we also saw (amongst others). And also in our case those bots were executing the JS, even hitting our Google Analytics. For some reason GA reported this traffic to come from Peru and Philippines, while an IP lookup showed it belonged to FB registered in the US or Ireland

Re: A Facebook crawler was making 7M requests per day to my stupid website

#364

Post by cranekam »

gk1 wrote: post #175
cranekam wrote: post #146

Earlier quoted context omitted.

FB has 2a00::/12 (and probably other blocks?) and can make something that looks like its company name from [0-9a-f]. Why wouldn't it do something harmless and fun like this? It's not as if it requires special dispensation or is breaking any rules.

I don't think the person was implying it's bad. Just pointing out an easter egg.

Ah, well if that’s the case then I apologise. My bad!

Re: A Facebook crawler was making 7M requests per day to my stupid website

#365

Post by lun4r »

wheresmycraisin wrote: post #261
lun4r wrote: post #29

We've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (used "spoofed" UAs) but were all coming from FB owned netblocks. I've contacted FB about it, but they couldn't figure out why this was happening and didn't solve the problem. I found out that there is an option in the FB Catalog manager that lets FB auto-remove…

Side question: how did you get in contact with facebook? I've an ad account that was suspended last year and gave up trying to contact them.

I have an option to email them, live chat, or have our account manager contact us. I guess if you spend enough on ads per month you are entitled to more support...?

Re: A Facebook crawler was making 7M requests per day to my stupid website

#366

Post by cranekam »

detaro wrote: post #210
cranekam wrote: post #146

Earlier quoted context omitted.

FB has 2a00::/12 (and probably other blocks?) and can make something that looks like its company name from [0-9a-f]. Why wouldn't it do something harmless and fun like this? It's not as if it requires special dispensation or is breaking any rules.

A single company does not get a /12 prefix. 2a00::/12 is almost half of the space currently allocated to all of RIPE NCC. Facebook seems to have 2a03:2880::/29 out of that /12, and a /40 through ARIN (2620:0:1c00::/40)

Ugh right you are. I misread the whois output and didn't stop to consider how realistic a /12 was. This is embarrassing.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#367

Post by albatross »

adrianN wrote: post #338
lun4r wrote: post #29

We've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (used "spoofed" UAs) but were all coming from FB owned netblocks. I've contacted FB about it, but they couldn't figure out why this was happening and didn't solve the problem. I found out that there is an option in the FB Catalog manager that lets FB auto-remove…

Set up a robots.txt that disallows Facebook crawlers, sue Facebook if the crawling continues for unauthorized access to computer systems, profit.

I believe this only works if it is coming from a corporation and targeted at an individual -_-

Re: A Facebook crawler was making 7M requests per day to my stupid website

#368

Post by Thorrez »

golem14 wrote: post #286
golem14 wrote: post #276

Earlier quoted context omitted.

That was the idea. There's a name for it - zip bomb. https://en.wikipedia.org/wiki/Zip_bomb A superficial search leads to things like https://www.rapid7.com/db/modules/auxiliary/dos/http/gzip_bo... https://stackoverflow.com/questions/1459673 You really want to be careful about potentially breaking laws ...

Here is another article: https://www.blackhat.com/docs/us-16/materials/us-16-Marie-I-... If FB supports brotli, a much bigger compression factor than 1000 is possible, apparently.

Here's a brotli file I created that's 81MB compressed and 100TB uncomrpessed[1] (bomb.br). That's a 1.2M:1 compression ratio (higher than any other brotli ratio I see mentioned online).

There's also a script in that directory that allows you to create files of whatever size you want (hovering around that same compression ratio). You can even use it to embed secret messages in the brotli (compressed or uncompressed). There's also a python script there that will serve it with the right header. Note that for Firefox it needs to be hosted on https, because Firefox only supports brotli over https.

Back when I created it, it would crash the entire browser of ESR Firefox, crash the tab of Chrome, and would lead to a perpetually loading page in regular Firefox.

It's currently hosted at [CAREFUL] https://stuffed.web.ctfcompetition.com [CAREFUL]

[1] https://github.com/google/google-ctf/tree/master/2019/finals...

Re: A Facebook crawler was making 7M requests per day to my stupid website

#369

Post by peanutz454 »

corobo wrote: post #321
redeux wrote: post #278

Earlier quoted context omitted.

In my experience they're one of the most useless and difficult companies I've ever tried to interact with.

The entirety of the universe is contained in the preceding two comments.

Aah, finally it all makes sense!

Re: A Facebook crawler was making 7M requests per day to my stupid website

#370

Post by inopinatus »

zdragnar wrote: post #357
inopinatus wrote: post #350

Earlier quoted context omitted.

Principled neutrality is fine for acceptable use. There’s no moral quandary in closing the door to abusers.

Isn't that the argument that providers make for wanting to meter usage? I.e. video streamers, torrenters and netflix and the like are 'abusing' the network by using a disproportionate amount of their capacity / bandwidth? I guess my point is that "abuse" in this sense is pretty subjective.

Not really, those things are easy to contrast. Network providers have always been comfortable blackholing DoS routes, and it’s never been controversial. That’s clearly distinct from those wanting to double-dip on transport revenues for routine traffic.

The difference is in whether both endpoints want the traffic, not whether (or on what basis) the enabling infrastructure wants to bear it.

Post reply on HN