Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Facebook crawler was making 7M requests per day to my stupid website

coding.napolux.com

Post reply on HN
Print view View original

201–210 of 416 posts

Re: A Facebook crawler was making 7M requests per day to my stupid website

#201

Post by Dahoon »

jimmaswell wrote: post #177
mastermojo wrote: post #160

Earlier quoted context omitted.

The author probably cares about either copy-left or the right to tinker. Some people believe it's important and others don't. Basically any derivative product out of GPL3 code needs to either be open-source/copy-left, or if it uses the code as a library needs to let end users substitute that library for their own version.

Worse than unimportant, I find copyleft harmful to the developer community as a whole. Software is hardly "free" when you have to bend to the author's demands to license your entire rest of your project under their ideology if you want to use it, and I think we should stop calling it so. Maybe "Conditionally Free Software" instead. You're hardly helping the world releasing libraries in terms that nobody but hobbyists…

You sound very bitter that a developers that let others use their code gets to pick his license of choice. Should everything by locked down like Microsoft Windows code or an Apple phone? If what you want is for everything to be completely free from licenses instead, just code your own version and release it as freeware. If you only complain and don't then you are just being hypocritical.

>You're hardly helping the world releasing libraries in terms that nobody but hobbyists (who are going to rip your code out and replace it if they start making a proprietary product, which is not evil) will find acceptable

Now you are just trolling.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#202

Post by Dahoon »

johnwheeler wrote: post #165
ruuda wrote: post #140

Hi Napolux, It looks like your site is using a theme based on my website ( https://ruudvanasseldonk.com/ , source at https://github.com/ruuda/blog ). That is fine — it is open source after all, licensed under the GPLv3. But I can’t find the source code for your site, and I can’t find any prominent notices saying that you modified my source. Could you please add those?

Next time, have some courtesy for the author and the rest of us by requesting via personal exchange over email instead of hijacking the thread and distracting from the conversation.

> ... and the rest of us ...

Please don't try to police the thread and speak for yourself only.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#203

Post by yev »

thelean12 wrote: post #172
ruuda wrote: post #163

Earlier quoted context omitted.

I checked the source and it does indeed mention https://wordpress.org/themes/blankslate/ , but judging from the screenshot there, that theme is just really a blank theme with no style at all, and it was used to include a different stylesheet. The real style is at https://coding.napolux.com/wp-content/themes/coding.napolux.... , which looks like normalize.css followed by a Wordpress adaptation of my stylesheet. The si…

Come on, this is super petty. And that includes your license choice and your enforcement for this.

I would back you up on this. This is super petty to be like that over a couple of lines of css.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#204

Post by [deleted] »

ThrustVectoring wrote: post #199
PetahNZ wrote: post #161

Earlier quoted context omitted.

My understanding is unless he's distributing it (not just serving it from a website) he doesnt need to release his changes. That's what the AGPL is for.

It's not the copyright on the server code that is at issue, but the copyright on the HTML and CSS files (and portions thereof) that get distributed by the server.

[deleted]

Re: A Facebook crawler was making 7M requests per day to my stupid website

#205

Post by nickysielicki »

lun4r wrote: post #54
lun4r wrote: post #51

Related thread at SO: https://stackoverflow.com/questions/49577546/facebook-crawle...

And FB bug report: https://developers.facebook.com/support/bugs/189402442061080...

It bugs me that they require a login to see a bug report, especially in the case of this bug where those affected aren't necessarily facebook users.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#206

Post by dahfizz »

jimmaswell wrote: post #193
Drakim wrote: post #185

Earlier quoted context omitted.

Copyleft is about maximizing the users's rights, not the developers's rights. Companies can't take linux, put it on a router for sale, and then say that their customers/users aren't allowed to know what's going on on the box in terms of backdoors and spying. The users have a right to look at the source code if they wish.

Users don't care about any of that, but developers are certainly hindered by restrictive licenses, which in turn hurts users. I never found a right to see source code compelling as a real right. To read the assembly and modify something they bought, sure, but not an entitlement to the source.

Nobody is forcing you to use GPL code. It is stinking of entitlement to demand that you get to use other people's code regardless of what they think about it.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#207

Post by djxfade »

golem14 wrote: post #198
napolux wrote: post #132

Earlier quoted context omitted.

I think we're together in this, my friend

I believe, but have not tried, that you can craft zipped files that are small but when expanding produce multi-gigabyte files. If you can sufficiently target the bad guys, you can probably grind them to a halt by serving only them these files. Some care needs to be taken so you don’t harm the innocent and don’t run afoul of any laws that may apply. Good luck.

Do you think it would be possible to create something similar for gzip? If you then serve with Content-Type: text/html, and Content-Encoding: gzip, the client would accept the payload. And when it tries to expand it, it would get expanded to a large file, eating up their resources.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#209

Post by jrockway »

dgellow wrote: post #170
jrockway wrote: post #134

Earlier quoted context omitted.

Shouldn't any public facing website have a rate limit? If someone attempts to circumvent simple rate limits (randomizing the source IP or header content), then that could demonstrate intent to cause damage, and you'd have a better case. But if you don't set a limit, how can you be mad that someone exceeded it? (I know they're ignoring robots.txt, but robots.txt is not a law. And, it doesn't apply to user-generated re…

The author specified that the crawler ignores 429 status code. So they do have some rate limit.

I noticed that, but I get the feeling that sending that status code didn't reduce the amount of work the program did, so it's kind of a pointless. You can say "go away", but at some point, you probably have to make them go away. (All RFC6585 says about 429 is that it MUST NOT be cached, not that the user agent can't try the request again immediately. It is very poorly specified.)

The author mentioned that they were using Cloudflare, and I see at least two ways to implement rate limiting. One is to have your application recognize the pattern of malicious behavior, and instruct Cloudflare via its API to block the IP address. Another is to use their rate-limiting that detects and blocks this stuff automatically (so they claim).

Like, there are many problems here. One is a rate limit that doesn't reduce load enough to serve legitimate requests. Another is wanting a caching service to cache a response that "MUST NOT" be cached. And the last is expecting people on the Internet to behave. There are always going to be broken clients, and you probably want the infrastructure in place to flat-out stop replying to TCP SYN packets from broken/malicious networks for a period of time. If the SYN packets themselves are overwhelming, welp, that is exactly why Cloudflare exists ;)

The author is right to be mad at Facebook. Their program is clearly broken. But it's up to you to mitigate your own stuff. This time it's a big company with lots of money that you can sue. The next time it will be some script kiddie in some faraway country with no laws. Your site will be broken for your users in either case, and so it falls on you to mitigate it.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#210

Post by detaro »

cranekam wrote: post #146
rcfox wrote: post #111

Did anyone notice the branded IP address? 2a03:2880:20ff:d::face:b00c "face:b00c"

FB has 2a00::/12 (and probably other blocks?) and can make something that looks like its company name from [0-9a-f]. Why wouldn't it do something harmless and fun like this? It's not as if it requires special dispensation or is breaking any rules.

A single company does not get a /12 prefix. 2a00::/12 is almost half of the space currently allocated to all of RIPE NCC. Facebook seems to have 2a03:2880::/29 out of that /12, and a /40 through ARIN (2620:0:1c00::/40)
Post reply on HN