Earlier quoted context omitted.
instead of scraping then, they could pay the fsf for a dump of the site or some API access or something, right? why overload the servers normally.
That's what commoncrawl does.
common crawl pays the sites they crawl?
How the FSF sysadmins block botnets with reaction
11–20 of 87 posts
Re: How the FSF sysadmins block botnets with reaction
#12> This software is gay, trans and anticolonialist. If you're uncomfortable with that, please don't use it
Weird message to include in AGPLv3 licensed software (which explicitly allows people to use software however they like, regardless of their beliefs or feelings).
Re: How the FSF sysadmins block botnets with reaction
#13> We placed our regular expressions in fail2ban, and found that we were hitting the maximum rules that could be added to UFW firewall rules on our systems which showed degradation around 65,000 rules Firewalld had a similar issue up until recently as well.
Re: How the FSF sysadmins block botnets with reaction
#14> This software is gay, trans and anticolonialist. If you're uncomfortable with that, please don't use it Weird message to include in AGPLv3 licensed software (which explicitly allows people to use software however they like, regardless of their beliefs or feelings).
You can have preferences while not restricting legal rights.
Re: How the FSF sysadmins block botnets with reaction
#15My personal preference is to 'ip route add blackhole ${net}' as it has the lowest CPU overhead and I can add hundreds of thousands of CIDR blocks with no noticeable impact. The only downside is that it won't stop UDP packets from getting to a UDP listener. There will not be a response but the application will still see it. For my TCP daemons it's great.
grep -m1 -E ^Tot /proc/net/fib_triestat ;ip route | grep -Fc blackhole
Total size: 56735 kB
426951
I also use a couple generalized rules in nftables raw table that keeps a lot of beyond poorly written bots away including hping3 tcp floods and masscan. My rules to port 443 are stateless. One must not taunt the state table.
Re: How the FSF sysadmins block botnets with reaction
#16It's somewhat interesting to see the FSF's approach to this. From what I understand they can't really use something like anubis since they want their websites to be accessible without javascript:
https://www.gnu.org/philosophy/javascript-trap.html
Users can't consent to running a page's javascript the way they can consent to running a program they've intentionally downloaded, so it's effectively "non-free" regardless of license.
Re: How the FSF sysadmins block botnets with reaction
#17That's interesting. I haven't used fail2ban for a long time, but reaction is worth evaluating. Unfortunately, that post does not describe their full configuration. Maybe it's on purpose, so that attackers can't adjust to fit.
My experience is that modern web scraping had no obvious pattern, since it is proxied through many IPs. The last time a server was failing to handle the pressure, we decided to temporarily ban IPs from some Asian regions. How does the FSF decide to ban an IP?
Why do they use iptables + ipset instead of nftables? Is there a technical reason or is it just legacy? AFAIK, Nftables is more performant, and IMO simpler. And it has native sets, see https://wiki.nftables.org/wiki-nftables/index.php/Sets
Re: How the FSF sysadmins block botnets with reaction
#18It's somewhat interesting to see the FSF's approach to this. From what I understand they can't really use something like anubis since they want their websites to be accessible without javascript: https://www.gnu.org/philosophy/javascript-trap.html Users can't consent to running a page's javascript the way they can consent to running a program they've intentionally downloaded, so it's effectively "non-free" regardless…
Anubis does support the no-JS HTTP meta-redirect proof of work but few know about it and fewer enable it. And it may not block everything.
Re: How the FSF sysadmins block botnets with reaction
#19It's somewhat interesting to see the FSF's approach to this. From what I understand they can't really use something like anubis since they want their websites to be accessible without javascript: https://www.gnu.org/philosophy/javascript-trap.html Users can't consent to running a page's javascript the way they can consent to running a program they've intentionally downloaded, so it's effectively "non-free" regardless…
Anubis does support the no-JS HTTP meta-redirect proof of work but few know about it and fewer enable it. And it may not block everything .
I indeed did not know about this. There seem to be some caveats:
https://anubis.techaro.lol/docs/admin/configuration/challeng...
I guess for the meta refresh challenge it's less "proof of work" and more "proof of patience".
Re: How the FSF sysadmins block botnets with reaction
#20> This software is gay, trans and anticolonialist. If you're uncomfortable with that, please don't use it Weird message to include in AGPLv3 licensed software (which explicitly allows people to use software however they like, regardless of their beliefs or feelings).
You can have preferences while not restricting legal rights.
You can, but if the exact quote in the GP is correct the claim is claiming the software is "gay, trans and anti-colonialist" and asks you not to use it. Why use a license that is designed to be politically neutral and then ask some people not to use it?
What I can see is a fairly clear indication that they do not want contributions from people whose politics differ from theirs. I would also question whether government funding of a project with political policies about who can participate is appropriate. The political stance is also rooted in a particular culture so is unwelcoming to people from other cultures.
Of course people can political views and preferences, but they presumably have some aim in mind when making that statement in the README. What is that aim?