Live data from Hacker News

A Facebook crawler was making 7M requests per day to my stupid website

coding.napolux.com

321–330 of 416 posts

Re: A Facebook crawler was making 7M requests per day to my stupid website

#321
post #278

Earlier quoted context omitted.

Really? I've never had problems contacting them by email. They're one of the easiest tech companies to talk to.

In my experience they're one of the most useless and difficult companies I've ever tried to interact with.

The entirety of the universe is contained in the preceding two comments.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#322
post #168
post #29

We've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (used "spoofed" UAs) but were all coming from FB owned netblocks. I've contacted FB about it, but they couldn't figure out why this was happening and didn't solve the problem. I found out that there is an option in the FB Catalog manager that lets FB auto-remove…

I just had an idea: if you control your own name server I believe you could use a BIND view to send all their own traffic to themselves based on the source address. By the way, if someone discovers how to trigger this issue it would be easy to use it as a DOS pseudo-botnet.

Something about this idea sits uncomfortably with me. I also just had an idea / thought experiment based on your idea.

We think of net neutrality as being for carriers and ISPs, but you could see it applied to a publicly accessible DNS service too. These DNS service providers are just as much part of the core service of the Internet as anyone else. It’s not a huge leap to require that those who operate a publicly accessible DNS service are bound by the same spirit of the regulations: that the infrastructure must not discriminate based on who is using it.

It’s different to operating a discriminatory firewall. DNS is a cacheable public service with bad consequences if poisonous data ends up in the system. Fiddling with DNS like this doesn’t seem like a good idea. Too much weird and bad stuff could go wrong.

Another analogy would be to the use of encryption on amateur radio. It seems like an innocuously good idea, but the radio waves were held open in public trust for public use. If you let them be used for a different (though arguably a more useful purpose) then the resource ends up being degraded.

Also along these lines of thought [begin irony mode]: FCC fines for DNS wildcard abuse / usage.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#323

Earlier quoted context omitted.

Don't think I used my email for anything important doing my time at FB. If it gets out of hand he could just make a request to have a new primary email made and use the above one for "spam"

Curiousity question: does FB use Gmail/Google suite?

My impression is that they pretty much roll their own communication suite.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#324
post #210

Earlier quoted context omitted.

A single company does not get a /12 prefix. 2a00::/12 is almost half of the space currently allocated to all of RIPE NCC. Facebook seems to have 2a03:2880::/29 out of that /12, and a /40 through ARIN (2620:0:1c00::/40)

ugh why is ipv6 impossible to understand :/

It's not. Think IPv4 but bigger.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#325

I'm new to webadmining in the cloud and my website is getting hammered by baidu, google, fb, et others causing traffic I/O costs to increase. What's an AWS LoadBalancer way of blocking this traffic? Again, noob here. THanks.

Don't host this stuff on AWS if you care about cost.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#326
Facebook caches URLs pretty aggressively (often you need to explicitly purge the cache to pick up updates) so I don't quite understand exactly what happened here. The article is very light on details. Do you have 7 million unique URLs that have been shared on Facebook? Or is it the same URL being scraped over and over?

Are you correctly sending caching headers?

Re: A Facebook crawler was making 7M requests per day to my stupid website

#327

Earlier quoted context omitted.

Don't think I used my email for anything important doing my time at FB. If it gets out of hand he could just make a request to have a new primary email made and use the above one for "spam"

Curiousity question: does FB use Gmail/Google suite?

FB uses Office365 for email. It was on-premise Exchange many many years ago, but moved "to the cloud" a while back.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#329
post #201

Earlier quoted context omitted.

You sound very bitter that a developers that let others use their code gets to pick his license of choice. Should everything by locked down like Microsoft Windows code or an Apple phone? If what you want is for everything to be completely free from licenses instead, just code your own version and release it as freeware. If you only complain and don't then you are just being hypocritical. >You're hardly helping the wo…

I'm not bitter, just disappointed in all the wasted developer time that happens because people get caught up in these copyleft ideas. Of course you can pick whatever license you want but that doesn't mean it can't be criticized. > If you only complain and don't then you are just being hypocritical. You don't need to be an architect to complain about crumbling bridges, but indeed I have released software under more fr…

> people get caught up in these copyleft ideas

Because of these ideas, we have nice things like open source systems and such.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#330
post #29

We've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (used "spoofed" UAs) but were all coming from FB owned netblocks. I've contacted FB about it, but they couldn't figure out why this was happening and didn't solve the problem. I found out that there is an option in the FB Catalog manager that lets FB auto-remove…

>> The bots didn't identify as FB (used "spoofed" UAs) That's surprising. What were the spoofed user agents that they used? We've run into this issue also, but all Facebook bot activity had user agents that contained the string "facebookexternalhit".

I've seen these these two user-agents from FB IPs, maybe others:

Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53

Mozilla/5.0 (iPhone; CPU iPhone OS 13_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Mobile/15E148 Safari/604.1

Which also execute javascript in some modified sandbox or something, causing errors, and executing error handlers. Interesting attempt to analyze the crawler here: https://github.com/aFarkas/lazysizes/issues/520#issuecomment...

Post reply on HN