Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Facebook crawler was making 7M requests per day to my stupid website

coding.napolux.com

Post reply on HN
Print view View original

311–320 of 416 posts

Re: A Facebook crawler was making 7M requests per day to my stupid website

#311

Post by laughinghan »

yjftsjthsd-h wrote: post #266
detaro wrote: post #241

Earlier quoted context omitted.

Possibly trying to avoid people sending them a different version of the page than users would see (of course they could change the page after the initial caching of a preview, but Twitter might refresh/check them later). Also, you often need an impressive amount of the stuff thats in a normal UA string for random sites to not break/send you the "unsupported browser, please use Netscape 4 or newer!!!" page/..., althou…

Yeah, I once tried to tell my browser to send... I forget; either no UA, or a blank UA string, or nonsense or just "Firefox" or something. I figured, "hey, some sites might break, but it can't be that important!" It broke everything . IIRC, the breaking point was that my own server refused to talk to me. Now, I still think this is insane, but apparently this really is how it is right now.

That's amazing. Any idea what piece of middleware on your own server was doing that?

Re: A Facebook crawler was making 7M requests per day to my stupid website

#312

Post by vvG94KbDUtRa »

detaro wrote: post #210
cranekam wrote: post #146

Earlier quoted context omitted.

FB has 2a00::/12 (and probably other blocks?) and can make something that looks like its company name from [0-9a-f]. Why wouldn't it do something harmless and fun like this? It's not as if it requires special dispensation or is breaking any rules.

A single company does not get a /12 prefix. 2a00::/12 is almost half of the space currently allocated to all of RIPE NCC. Facebook seems to have 2a03:2880::/29 out of that /12, and a /40 through ARIN (2620:0:1c00::/40)

ugh why is ipv6 impossible to understand :/

Re: A Facebook crawler was making 7M requests per day to my stupid website

#313

Post by licebmi__at__ »

hombre_fatal wrote: post #303
yjftsjthsd-h wrote: post #266

Earlier quoted context omitted.

Yeah, I once tried to tell my browser to send... I forget; either no UA, or a blank UA string, or nonsense or just "Firefox" or something. I figured, "hey, some sites might break, but it can't be that important!" It broke everything . IIRC, the breaking point was that my own server refused to talk to me. Now, I still think this is insane, but apparently this really is how it is right now.

Should make you realize just how much abuse there is on the internet that it's worth it to just filter traffic with no UA. Usually people get stuck on the fact that we can't have nice things, so X sucks for not letting us have nice things, yet I seem to never see people acknowledge why we can't have nice things. Then I'd see a lot more "ugh, bad actors suck!" and less "ugh, websites are just trying to make life miser…

Well, if filtering for UA really makes things difficult for bad actors, they do suck but more in as a technical opinion than an moral statement.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#314

Post by jasonlotito »

thelean12 wrote: post #186
jasonlotito wrote: post #183

Earlier quoted context omitted.

So, just to be clear, you are perfectly fine with someone taking something someone else created and violating the terms upon which they were given that thing? e.g. If I took code you wrote and lets say released under an MIT license and claimed I wrote it and didn't give you any credit, and in fact released it under another license entirely, you'd be fine with that?

It's perfectly valid to criticize the original license choice. GPLv3 is a very restrictive license, especially for what is essentially a micro blog (though I dislike the license for most open source software anyway). Add on the original author going after a bit of CSS, not even the main effort of the project in question, and you've got my "petty" comment.

You didn't answer my question. That pretty much says all that needs to be said.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#315

Post by jasonlotito »

Aeolun wrote: post #245
jasonlotito wrote: post #183

Earlier quoted context omitted.

So, just to be clear, you are perfectly fine with someone taking something someone else created and violating the terms upon which they were given that thing? e.g. If I took code you wrote and lets say released under an MIT license and claimed I wrote it and didn't give you any credit, and in fact released it under another license entirely, you'd be fine with that?

> e.g. If I took code you wrote and lets say released under an MIT license and claimed I wrote it and didn't give you any credit, and in fact released it under another license entirely, you'd be fine with that? If I released it on Github, under any license whatever? I’d more or less be expecting that. If it was about the 4hr of work that went into my blog theme, I wouldn’t be bothered at all . But then, I wouldn’t re…

So, you are okay with people violating other peoples licenses and ignoring copyright. Gotcha.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#316

Post by deadwing0 »

missedthecue wrote: post #271
wheresmycraisin wrote: post #261

Earlier quoted context omitted.

Side question: how did you get in contact with facebook? I've an ad account that was suspended last year and gave up trying to contact them.

Really? I've never had problems contacting them by email. They're one of the easiest tech companies to talk to.

What did you contact them for? Just curious as I've almost always heard they're like Google and impossible to get a human response from.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#317

Post by laughinghan »

ruuda wrote: post #140

Hi Napolux, It looks like your site is using a theme based on my website ( https://ruudvanasseldonk.com/ , source at https://github.com/ruuda/blog ). That is fine — it is open source after all, licensed under the GPLv3. But I can’t find the source code for your site, and I can’t find any prominent notices saying that you modified my source. Could you please add those?

Your blog is awesome! I'm really glad you commented here, or I wouldn't have found it. I even forwarded your most recent post to some friends.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#318

Post by foobiekr »

pera wrote: post #168
lun4r wrote: post #29

We've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (used "spoofed" UAs) but were all coming from FB owned netblocks. I've contacted FB about it, but they couldn't figure out why this was happening and didn't solve the problem. I found out that there is an option in the FB Catalog manager that lets FB auto-remove…

I just had an idea: if you control your own name server I believe you could use a BIND view to send all their own traffic to themselves based on the source address. By the way, if someone discovers how to trigger this issue it would be easy to use it as a DOS pseudo-botnet.

if only there was a way to accidentally amplify that.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#319

Post by thenickdude »

hank_z wrote: post #306

I’m a bit confused. Facebook is social media company, why do they send out crawlers?

Facebook shows previews of pretty much everything you share a link to, so at a minimum they have to fetch the page/image to build those previews.
Post reply on HN