Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Facebook crawler was making 7M requests per day to my stupid website

coding.napolux.com

Post reply on HN
Print view View original

351–360 of 416 posts

Re: A Facebook crawler was making 7M requests per day to my stupid website

#351

Post by archon810 »

dang wrote: post #235
apolymath wrote: post #230

Earlier quoted context omitted.

Flattered that your few, boring lines of CSS and HTML (that could easily be reproduced by a monkey) was used on someone else's website?

We've banned this account for breaking the site guidelines. If you don't want to be banned, you're welcome to email hn@ycombinator.com and give us reason to believe that you'll follow the rules in the future. https://news.ycombinator.com/newsguidelines.html

Toxic comments have no place on HN. Thank you for policing them so diligently.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#352

Post by Ueland »

cdown wrote: post #89
napolux wrote: post #65

Earlier quoted context omitted.

Thanks man! I'll have a look.

Hey! Facebook engineer here. If you have it, can you send me the User-Agent for these requests? That would definitely help speed up narrowing down what's happening here. If you can provide me the hostname being requested in the Host header, that would be great too. I just sent you an e-mail, you can also reply to that instead if you prefer not to share those details here. :-)

Hey,

I can by the way confirm this issue. I work in a large newspaper in Norway and around a year ago we saw the same issue. Thousands of requests per second until we blocked it . And after we blocked it, traffic to our Facebook page also plummeted. I assume Facebook considered our website down and thus wouldn't give users content from our Facebook page either as that would serve them content that would give a bad user experience. The Facebook traffic did not normalize before the attack stopped AND after we told Facebook to reindex all our content.

I'd you want more info, send me a email and il dig out some logs etc. thu at db.no

Re: A Facebook crawler was making 7M requests per day to my stupid website

#354

Post by namibj »

staycoolboy wrote: post #333
namibj wrote: post #325

Earlier quoted context omitted.

Don't host this stuff on AWS if you care about cost.

So you don't know how to do this in AWS is what I'm hearing?

My point is, that you don't want to do this inside of a load balancer there. I don't recall any traffic filtering abilities that would suffice, but I'm not fully up-to-date with the configurability either. If the load balancer supports it, a short search in the net or docs should surface an easily-applicable guide, and if not, I'd probably put that blocking closer to my app server.

And the reason against AWS for this would be both the general cost (AWS is not cost-efficient in many cases, unless you have complicated infrastructure that takes a lot of management, where the Infrastructure-as-code approach can give you a sizable benefit), the bandwidth cost in particular, and the lack of configurability of their services to e.g. apply suitable tar-pitting against such crawlers.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#355

Post by 6c696e7578 »

lun4r wrote: post #29

We've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (used "spoofed" UAs) but were all coming from FB owned netblocks. I've contacted FB about it, but they couldn't figure out why this was happening and didn't solve the problem. I found out that there is an option in the FB Catalog manager that lets FB auto-remove…

Thank goodness for mod_rewrite, which makes blocking/redirecting traffic on basic things like headers pretty easy.

https://www.usenix.org.uk/content/rewritemap.html

You could of course block upstream by IP, but if you want to send the traffic away from a CPU heavy dynamic page to something static that 2xx's or 301's to https://developers.facebook.com/docs/sharing/webmasters/craw... then this could be the answer.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#357

Post by zdragnar »

inopinatus wrote: post #350
gorgoiler wrote: post #322

Earlier quoted context omitted.

Something about this idea sits uncomfortably with me. I also just had an idea / thought experiment based on your idea. We think of net neutrality as being for carriers and ISPs, but you could see it applied to a publicly accessible DNS service too. These DNS service providers are just as much part of the core service of the Internet as anyone else. It’s not a huge leap to require that those who operate a publicly acc…

Principled neutrality is fine for acceptable use. There’s no moral quandary in closing the door to abusers.

Isn't that the argument that providers make for wanting to meter usage? I.e. video streamers, torrenters and netflix and the like are 'abusing' the network by using a disproportionate amount of their capacity / bandwidth?

I guess my point is that "abuse" in this sense is pretty subjective.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#358

Post by auscompgeek »

lkrubner wrote: post #349
hombre_fatal wrote: post #303

Earlier quoted context omitted.

Should make you realize just how much abuse there is on the internet that it's worth it to just filter traffic with no UA. Usually people get stuck on the fact that we can't have nice things, so X sucks for not letting us have nice things, yet I seem to never see people acknowledge why we can't have nice things. Then I'd see a lot more "ugh, bad actors suck!" and less "ugh, websites are just trying to make life miser…

Is there an official error message for that? Because filtering no UA would trip me up every time I use wget. Most of the time, if I'm casually using wget for something, I don't bother with a UA. If sites started rejecting that, I'd like to get a clear error message, so I would not go crazy trying to figure out what the problem was. If I got a clear message "send a UA" then I would probably started wrapping my wget re…

wget has a default User-Agent string.
Post reply on HN