Live data from Hacker News

A Facebook crawler was making 7M requests per day to my stupid website

coding.napolux.com

251–260 of 416 posts

Re: A Facebook crawler was making 7M requests per day to my stupid website

#251

Earlier quoted context omitted.

Come on. I love Ruud's posts and I took inspiration for my blog. He was right asking for a link, which I gladly provided. That's it for me.

Sorry, I think you might have misinterpreted my comment?

It seems he intended to reply to the parent comment.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#252
post #72

Earlier quoted context omitted.

IP packets have a source field, which can be fake and not their actual IP. That's a Facebook IP, but the packet might not have actually come from it.

UDP packets can fake their source IP. TCP packets realistically can't.

More precisely, spoofing a TCP handshake is a problem that we know how to prevent. Odds are pretty good that your kernel has these protections enabled by default, but it's not guaranteed and you should check as a matter of due diligence.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#253
post #251

Earlier quoted context omitted.

Sorry, I think you might have misinterpreted my comment?

It seems he intended to reply to the parent comment.

Ah, yes, I see that now. Thanks for clearing that up.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#254
post #108

That's 81 request per second on average. Shouldn't anybody doing such thing be liable, and be sued for negligence and required to pay damages? That sounds like a lot of bandwidth (and server stress).

Yes, this was my first thought too - Facebook is essentially denial of servicing people's sites and it sounds like they're aware of it. Move fast and break your things. If this were your average Joe I'm fairly certain something like the DMCA could be used.

What? How does the DCMA apply here at all?

Re: A Facebook crawler was making 7M requests per day to my stupid website

#255
post #197

Earlier quoted context omitted.

I never looked into the TARPIT option in iptables before reading your comment. That seems really useful. I've been dealing with on and off bursts of traffic from a single AWS region for the last month. They usually keep going for about 90 minutes every day, regardless of how many IPs I block, and consume every available resource with about 250 requests per second (not a big server and I'm still waiting for approval t…

Be careful, as tarpitting connections can consume your resources faster than those of the attacker.

Most spiders limit the number of requests per domain, so if it's stupidity and not malice, you probably don't have a runaway situation.

... unless you're hosting a lot of websites for people in a particular industry. In which case the bot will just start making requests to three other websites you also are responsible for.

Then if you use a tarpit machine instead of routing tricks, the resource pool is bounded by the capacity of that single machine. If you have 20 other machines that's just the Bad Bot Tax and you should pay it with a clean conscience and go solve problems your human customers actually care about.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#259
post #185

Earlier quoted context omitted.

Worse than unimportant, I find copyleft harmful to the developer community as a whole. Software is hardly "free" when you have to bend to the author's demands to license your entire rest of your project under their ideology if you want to use it, and I think we should stop calling it so. Maybe "Conditionally Free Software" instead. You're hardly helping the world releasing libraries in terms that nobody but hobbyists…

Copyleft is about maximizing the users's rights, not the developers's rights. Companies can't take linux, put it on a router for sale, and then say that their customers/users aren't allowed to know what's going on on the box in terms of backdoors and spying. The users have a right to look at the source code if they wish.

They can take Linux and put it behind a locked bootloader that makes it impossible to replace with your own kernel. Sure you can have the modified sources but you can't do anything with it. Hence GPL3.
Post reply on HN