Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Facebook crawler was making 7M requests per day to my stupid website

coding.napolux.com

Post reply on HN
Print view View original

301–310 of 416 posts

Re: A Facebook crawler was making 7M requests per day to my stupid website

#301

Post by sirclueless »

sprayk wrote: post #254
Traster wrote: post #120

Earlier quoted context omitted.

Yes, this was my first thought too - Facebook is essentially denial of servicing people's sites and it sounds like they're aware of it. Move fast and break your things. If this were your average Joe I'm fairly certain something like the DMCA could be used.

What? How does the DCMA apply here at all?

The DMCA has a couple major parts. The part most people are familiar with deals with safe harbor provisions and "DMCA takedowns". The relevant part for this discussion makes it illegal to circumvent digital access-control technology. So you could argue that if you have reasonable measures in place to stop automated access of your content such as robots.txt then Facebook continuing to send automated requests would be a circumvention of your access controls.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#302

Post by njvsifhgnvion »

djxfade wrote: post #207
golem14 wrote: post #198

Earlier quoted context omitted.

I believe, but have not tried, that you can craft zipped files that are small but when expanding produce multi-gigabyte files. If you can sufficiently target the bad guys, you can probably grind them to a halt by serving only them these files. Some care needs to be taken so you don’t harm the innocent and don’t run afoul of any laws that may apply. Good luck.

Do you think it would be possible to create something similar for gzip? If you then serve with Content-Type: text/html, and Content-Encoding: gzip, the client would accept the payload. And when it tries to expand it, it would get expanded to a large file, eating up their resources.

1 character repeated 1877302224 times has a compression ratio of 99.9% ~1.9GB compresses to ~1.8MB.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#303

Post by hombre_fatal »

yjftsjthsd-h wrote: post #266
detaro wrote: post #241

Earlier quoted context omitted.

Possibly trying to avoid people sending them a different version of the page than users would see (of course they could change the page after the initial caching of a preview, but Twitter might refresh/check them later). Also, you often need an impressive amount of the stuff thats in a normal UA string for random sites to not break/send you the "unsupported browser, please use Netscape 4 or newer!!!" page/..., althou…

Yeah, I once tried to tell my browser to send... I forget; either no UA, or a blank UA string, or nonsense or just "Firefox" or something. I figured, "hey, some sites might break, but it can't be that important!" It broke everything . IIRC, the breaking point was that my own server refused to talk to me. Now, I still think this is insane, but apparently this really is how it is right now.

Should make you realize just how much abuse there is on the internet that it's worth it to just filter traffic with no UA.

Usually people get stuck on the fact that we can't have nice things, so X sucks for not letting us have nice things, yet I seem to never see people acknowledge why we can't have nice things.

Then I'd see a lot more "ugh, bad actors suck!" and less "ugh, websites are just trying to make life miserable for me >:("

Re: A Facebook crawler was making 7M requests per day to my stupid website

#304

Post by technion »

rapind wrote: post #231
paulpauper wrote: post #190

Earlier quoted context omitted.

twitter does the same thing. it sends a bunch of spoofed US visitors from Korea and Germany and US. The bots are spoofed to make it harder to filter them.

I wonder what the (legitimate?) reason is for them to spoof. Seems intentionally shady. Maybe there's a legit reason we're missing?

When you know your target runs Internet Explorer, you serve phishing pages to IE users and some boring content to other users, at the server level. We've had this keep our user tests out of Google Safebrowse and so on. I'm sure similar tricks end up applied to Facebook's UA and "legitimate" marketing sites.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#305

Post by mulmen »

userbinator wrote: post #290
maxk42 wrote: post #195

Earlier quoted context omitted.

GPL license states you must provide source code to the end-user. In this case, the end-user is legally Napolux, not the visitors to his site.

In any case, View -> Source The source code of a public website is automatically provided to everyone who visits it. This fact is unfortunately not well-known.

That's... not how webpages work. This is why the AGPL even exists.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#308

Post by hombre_fatal »

stiray wrote: post #216
lun4r wrote: post #29

We've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (used "spoofed" UAs) but were all coming from FB owned netblocks. I've contacted FB about it, but they couldn't figure out why this was happening and didn't solve the problem. I found out that there is an option in the FB Catalog manager that lets FB auto-remove…

I dont really understand what is the issue. On my welcome page (while all other urls are impossible to guess) i give browser something that requires a few seconds of cpu at 100% to crunch. And tracking some user action in between, visting tarpitted urls etc. In last few years no bot came through. Why bother with robots.txt, just give them something to break their teeths... (I would give you the url, but I just dont w…

The far-fetched "solution" along with the "I don't understand everyone doesn't do " make it hard for me to believe you're actually doing this. Sounds more like a shower thought, maybe fun weekend project, than something in production.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#309

Post by thirtythree »

doommius wrote: post #104
traverseda wrote: post #99

Earlier quoted context omitted.

I'm not sure I'd publicly post my email like that, if I worked at FB. But congratulations on your promotion to "official technical contact for all facebook issues forever".

Don't think I used my email for anything important doing my time at FB. If it gets out of hand he could just make a request to have a new primary email made and use the above one for "spam"

Curiousity question: does FB use Gmail/Google suite?

Re: A Facebook crawler was making 7M requests per day to my stupid website

#310

Post by golem14 »

x0 wrote: post #294
golem14 wrote: post #198

Earlier quoted context omitted.

I believe, but have not tried, that you can craft zipped files that are small but when expanding produce multi-gigabyte files. If you can sufficiently target the bad guys, you can probably grind them to a halt by serving only them these files. Some care needs to be taken so you don’t harm the innocent and don’t run afoul of any laws that may apply. Good luck.

I believe zip bombs are pretty easily mitigated, with memory limits, cpu time ulimits, etc. But zip bombs aren't limited to zip files... lots of files have some compression in them You can make malicious PNG's that do the same thing. Probably tonnes of other files.

Good points. According to OP, the goal would be to persuade FB to back off crawling at 300 qps, not to bring down the FB crawl. Having each request expand to 1GB is probably already enough to do that, since a crawler implementing the mitigations you mentioned will likely reduce the crawl rate. Should be easily testable.
Post reply on HN