Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Facebook crawler was making 7M requests per day to my stupid website

coding.napolux.com

Post reply on HN
Print view View original

331–340 of 416 posts

Re: A Facebook crawler was making 7M requests per day to my stupid website

#331

Post by rock_hard »

callalex wrote: post #323
thirtythree wrote: post #309

Earlier quoted context omitted.

Curiousity question: does FB use Gmail/Google suite?

My impression is that they pretty much roll their own communication suite.

That’s somewhat correct

But at least for email/calendar backend its exchange

The internal replacement clients for calendar and other things are killer...have yet to find replacements

For the most part though they use Facebook internally for messaging and regular communication (technically now Worplace but before it was just Facebook)

Email is really just for external folks

Re: A Facebook crawler was making 7M requests per day to my stupid website

#332

Post by Talanes »

jimmaswell wrote: post #247
Dahoon wrote: post #201

Earlier quoted context omitted.

You sound very bitter that a developers that let others use their code gets to pick his license of choice. Should everything by locked down like Microsoft Windows code or an Apple phone? If what you want is for everything to be completely free from licenses instead, just code your own version and release it as freeware. If you only complain and don't then you are just being hypocritical. >You're hardly helping the wo…

I'm not bitter, just disappointed in all the wasted developer time that happens because people get caught up in these copyleft ideas. Of course you can pick whatever license you want but that doesn't mean it can't be criticized. > If you only complain and don't then you are just being hypocritical. You don't need to be an architect to complain about crumbling bridges, but indeed I have released software under more fr…

>wasted developer time

Only if you assume copyleft code would otherwise be available for developers. The other alternative is people who don't want others profiting from their free code just don't release their code.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#333

Post by staycoolboy »

namibj wrote: post #325
staycoolboy wrote: post #182

I'm new to webadmining in the cloud and my website is getting hammered by baidu, google, fb, et others causing traffic I/O costs to increase. What's an AWS LoadBalancer way of blocking this traffic? Again, noob here. THanks.

Don't host this stuff on AWS if you care about cost.

So you don't know how to do this in AWS is what I'm hearing?

Re: A Facebook crawler was making 7M requests per day to my stupid website

#334

Post by yjftsjthsd-h »

laughinghan wrote: post #311
yjftsjthsd-h wrote: post #266

Earlier quoted context omitted.

Yeah, I once tried to tell my browser to send... I forget; either no UA, or a blank UA string, or nonsense or just "Firefox" or something. I figured, "hey, some sites might break, but it can't be that important!" It broke everything . IIRC, the breaking point was that my own server refused to talk to me. Now, I still think this is insane, but apparently this really is how it is right now.

That's amazing. Any idea what piece of middleware on your own server was doing that?

I don't remember, but what really got me was that I wasn't running anything that I expected to do fancy filtering; I think this was just Apache httpd running on CentOS. But there was no web application firewall, no load balancers, pretty sure fail2ban was only set up for sshd. It at least appeared that just the apache stock config was in play.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#336

Post by Aeolun »

jasonlotito wrote: post #315
Aeolun wrote: post #245

Earlier quoted context omitted.

> e.g. If I took code you wrote and lets say released under an MIT license and claimed I wrote it and didn't give you any credit, and in fact released it under another license entirely, you'd be fine with that? If I released it on Github, under any license whatever? I’d more or less be expecting that. If it was about the 4hr of work that went into my blog theme, I wouldn’t be bothered at all . But then, I wouldn’t re…

So, you are okay with people violating other peoples licenses and ignoring copyright. Gotcha.

I just think it’s naive to assume they won’t.

It’s a bit like putting a solid gold bar on your lawn and putting a sign next to it saying ‘please don’t take, this is mine’.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#337

Post by licebmi__at__ »

jimmaswell wrote: post #247
Dahoon wrote: post #201

Earlier quoted context omitted.

You sound very bitter that a developers that let others use their code gets to pick his license of choice. Should everything by locked down like Microsoft Windows code or an Apple phone? If what you want is for everything to be completely free from licenses instead, just code your own version and release it as freeware. If you only complain and don't then you are just being hypocritical. >You're hardly helping the wo…

I'm not bitter, just disappointed in all the wasted developer time that happens because people get caught up in these copyleft ideas. Of course you can pick whatever license you want but that doesn't mean it can't be criticized. > If you only complain and don't then you are just being hypocritical. You don't need to be an architect to complain about crumbling bridges, but indeed I have released software under more fr…

Well, it's easy to be disappointed in all the wasted developer time because copy left. It's transparent.

It's harder not only to be disappointed but to even notice all the wasted developer time because of closed source code because well it's even obscured how can you as a developer benefit from the code.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#338

Post by adrianN »

lun4r wrote: post #29

We've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (used "spoofed" UAs) but were all coming from FB owned netblocks. I've contacted FB about it, but they couldn't figure out why this was happening and didn't solve the problem. I found out that there is an option in the FB Catalog manager that lets FB auto-remove…

Set up a robots.txt that disallows Facebook crawlers, sue Facebook if the crawling continues for unauthorized access to computer systems, profit.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#339

Post by omershapira »

I once made a PHP page which fetches a random wikipedia article, but changes the title to "What the fuck is [TOPIC]?". It was incredibly funny, until I started getting angry emails from people, mostly threatening a form of libel lawsuit.

Turns out, since it was a front for all of wikipedia[1], google was agressively indexing it, but the results rarely made it to the first search page. And since this isn't exactly an important site, old results would stick around.

Hence a pattern:

1. Some Rando creates a page about themselves

2. Wikipedia editors, being holy and good, extinguish that nonsense

1.5. GOOGLE INDEXES IT

3. Rando, by nature of being a rando, googles themselves, doesn't find their wikipedia page anymore (that's gone), but does find a link to my site in the first page of google results with their name.

4. Lawyers get involved, somehow

Details: http://omershapira.com/blog/2013/03/randomfax-net/

[1] Hebrew wikipedia. I can't imagine what would've happened on the English version.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#340

Post by thelean12 »

jasonlotito wrote: post #314
thelean12 wrote: post #186

Earlier quoted context omitted.

It's perfectly valid to criticize the original license choice. GPLv3 is a very restrictive license, especially for what is essentially a micro blog (though I dislike the license for most open source software anyway). Add on the original author going after a bit of CSS, not even the main effort of the project in question, and you've got my "petty" comment.

You didn't answer my question. That pretty much says all that needs to be said.

My answer is less useful to the discussion. But here you go:

1. I wouldn't use GPLv3

2. I wouldn't care if people stole my code that I open sourced or if they tried to license it a different way.

3. I personally follow the license of others when using their code. I wouldn't steal GPLv3 code without proper attribution etc. That's their right.

All that doesn't go against my initial opinion: GPLv3 for a small micro blog templating system is lame. Enforcing it for a bit of CSS is petty.

Post reply on HN