Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Facebook crawler was making 7M requests per day to my stupid website

coding.napolux.com

Post reply on HN
Print view View original

381–390 of 416 posts

Re: A Facebook crawler was making 7M requests per day to my stupid website

#381

Post by Nextgrid »

mszcz wrote: post #375
pera wrote: post #168

Earlier quoted context omitted.

I just had an idea: if you control your own name server I believe you could use a BIND view to send all their own traffic to themselves based on the source address. By the way, if someone discovers how to trigger this issue it would be easy to use it as a DOS pseudo-botnet.

I had a different idea. Maybe you could craft a zip-bomb response. The bot would fetch the small gzipped content and upon extraction discover it was GBs of data? Not sure that's possible here, when responding to a request, but that would surely turn the admins attention to it.

Here's an example of things you can do against malicious crawlers: http://www.hackerfactor.com/blog/index.php?/archives/762-Att....

Re: A Facebook crawler was making 7M requests per day to my stupid website

#382

Post by thaumasiotes »

dragonwriter wrote: post #36
thaumasiotes wrote: post #24

Earlier quoted context omitted.

Going against the HTTP standard isn't a problem. For example, it's a good practice to ignore HEAD requests as opposed to responding appropriately. The problem with unsafe GET is that it conflicts with reality, not that it conflicts with the standard.

> Going against the HTTP standard isn't a problem. For example, it's a good practice to ignore HEAD requests as opposed to responding appropriately. That's only against the standard of you advertise HEAD as a supported method on the resource, which converts it from a good idea in some circumstances to a bad one, so if there is a good example to support your claim, that isn't it.

The most classic example is the JWT specification, which says you need to honor the encryption algorithm defined by the token you receive. (JWT includes a "none" algorithm, making token forgery trivial when the parser implements the standard.)

It's known widely enough now that people have chosen to reinterpret the language of the standard in order to claim that their implementations are compliant -- after making the change specifically to bring themselves out of compliance.

(It's possible that it's been so long that the standard itself has been changed to accommodate this. But regardless, the point stands that standards compliance is not a virtue for its own sake. This wasn't a good idea back when everyone agreed that the standard required it, it's not a good idea now, and future bad ideas do not in general become good ideas by virtue of being specified in standards.)

Re: A Facebook crawler was making 7M requests per day to my stupid website

#384

Post by bishalb »

wheresmycraisin wrote: post #261
lun4r wrote: post #29

We've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (used "spoofed" UAs) but were all coming from FB owned netblocks. I've contacted FB about it, but they couldn't figure out why this was happening and didn't solve the problem. I found out that there is an option in the FB Catalog manager that lets FB auto-remove…

Side question: how did you get in contact with facebook? I've an ad account that was suspended last year and gave up trying to contact them.

Try their live chat.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#385

Post by justinph »

adrianN wrote: post #338
lun4r wrote: post #29

We've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (used "spoofed" UAs) but were all coming from FB owned netblocks. I've contacted FB about it, but they couldn't figure out why this was happening and didn't solve the problem. I found out that there is an option in the FB Catalog manager that lets FB auto-remove…

Set up a robots.txt that disallows Facebook crawlers, sue Facebook if the crawling continues for unauthorized access to computer systems, profit.

robots.txt is not a legal document. It is asking nicely, and plenty of crawlers purposefully ignore it.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#386

Post by touristtam »

Nextgrid wrote: post #381
mszcz wrote: post #375

Earlier quoted context omitted.

I had a different idea. Maybe you could craft a zip-bomb response. The bot would fetch the small gzipped content and upon extraction discover it was GBs of data? Not sure that's possible here, when responding to a request, but that would surely turn the admins attention to it.

Here's an example of things you can do against malicious crawlers: http://www.hackerfactor.com/blog/index.php?/archives/762-Att... .

fascinating read. :)

Re: A Facebook crawler was making 7M requests per day to my stupid website

#388

Post by adrianN »

justinph wrote: post #385
adrianN wrote: post #338

Earlier quoted context omitted.

Set up a robots.txt that disallows Facebook crawlers, sue Facebook if the crawling continues for unauthorized access to computer systems, profit.

robots.txt is not a legal document. It is asking nicely, and plenty of crawlers purposefully ignore it.

I'm not a lawyer, but I believe I remember people being sued for essentially making a GET request to a URL they weren't supposed to GET.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#389

Post by coronadisaster »

thelean12 wrote: post #340
jasonlotito wrote: post #314

Earlier quoted context omitted.

You didn't answer my question. That pretty much says all that needs to be said.

My answer is less useful to the discussion. But here you go: 1. I wouldn't use GPLv3 2. I wouldn't care if people stole my code that I open sourced or if they tried to license it a different way. 3. I personally follow the license of others when using their code. I wouldn't steal GPLv3 code without proper attribution etc. That's their right. All that doesn't go against my initial opinion: GPLv3 for a small micro blog…

> All that doesn't go against my initial opinion: GPLv3 for a small micro blog templating system is lame. Enforcing it for a bit of CSS is petty.

It is a matter of principle, probably?

Post reply on HN