Live data from Hacker News

Botspam apocalypse

memex.marginalia.nu

221–230 of 358 posts

Re: Botspam apocalypse

#221
post #2

> They're a major part in killing off web forums, and a significant wet blanket on any sort of fun internet creativity or experimentation. > The only ones that can survive the robot apocalypse is large web services. Your reddits, and facebooks, and twitters, and SaaS-comment fields, and discords. They have the economies of scale to develop viable countermeasures, to hire teams of people to work on the problem full ti…

> The solution is real simple Uhhmmm, I beg to differ and so do a lot of very smart people with many more servers and users than you or I are likely to see. As with most 'Oh, its' Simple - Just Do XYZ' solutions there are often very good reasons for not doing the 'Easy/Simple/One-Liner' and here are a few with yours - Firstly - The '10 bux' could exclude a vast swathe of the poorest. Skipping a couple of Starbuck cof…

[deleted]

Re: Botspam apocalypse

#222

It's annoying for sure. I deal with abuse at a large scale. I'd recommend: - Rate-limit everything, absolutely everything. Set sane limits. - Rate-limit POST requests harder. Preferably dynamically based on geoip. - Rate-limit login and comment POST requests even harder. Ban IPs that exceed the amount. - Require TLS. Drop TLSv1.0 and TLSv1.1. Bots certainly break. - Require SNI. Do not reply without SNI (nginx has 44…

> Rate-limit login and comment POST requests even harder. Ban IPs that exceed the amount

Don't ban IPs. Or if you do, let the ban expire relatively quickly (days/weeks, not months/years).

Re: Botspam apocalypse

#223

> If Marginalia Search didn't use Cloudflare, it couldn't serve traffic. Cloudflare is not the only CDN/protection. It's the most popular and the most evil one. You have a choice.

Why do you consider them to be "the most evil"? Their services seem to be completely fine in almost every regard, and their communication doesn't at all suggest that they might be evil.

Re: Botspam apocalypse

#224
post #192

Earlier quoted context omitted.

None of those are free, though.

If you're not paying, what's the product they're selling?

Their paid one when you go over the limits. Not everything has to be black and white and reduced down to a single, oft repeated catch phrase like that.

Re: Botspam apocalypse

#225
post #68

For small sites, I would just use a simple firewall: 1. whitelist the finite IP ranges for the regional ISPs/country where you do business 2. blacklist the proxy and tor exit nodes 3. blacklist the list of published compromised servers 4. add spamhaus blacklists 5. add fail2ban rules to trip on common server security scans, and unused common service ports 6. publicly reply to those having access issues, and imply the…

Tor users are often legitimate good internet citizens. A lot of (lucky) us have the luxury to live in real democracies. Some others live in countries that use every single aspect of their private lives (DPI, mass surveillance) to put pressure on them and bend them to the regime's will. In my opinion, Tor and anonymity should not be killed as a result of silly bots.

> Tor users are often legitimate good internet citizens.

We have had exactly zero traffic from it at any point which was legitimate. Any user who ever showed up with a exit IP ended up being banned eventually, so we just proactively fraud banned anybody who uses one, and anybody that was related to them. There is zero value in allowing anonymizer traffic on your service, and a whole lot to lose.

Re: Botspam apocalypse

#227
post #65

Earlier quoted context omitted.

>The solution is real simple: it costs 10 bux to register an account, if you're a nuisance your account is banned and you pay 10bux to get back on. Many years ago there was a public server called SDF (Super Dimensional Fortress). It was a BSD system and anyone could get a user account for $1. The theory was even the least of us, a kid scrounging for money on the street, could come up with a dollar (and presumably the…

> Many years ago there was a public server called SDF (Super Dimensional Fortress). It was a BSD system and anyone could get a user account for $1. The theory was even the least of us, a kid scrounging for money on the street, could come up with a dollar (and presumably the postage to mail it). To a certain person, access to this kind of server was invaluable - the only situation you could hope to get close to this k…

They even have a Mastodon instance.

https://mastodon.sdf.org/about

Re: Botspam apocalypse

#228

> If Marginalia Search didn't use Cloudflare, it couldn't serve traffic. There has been upwards of 15 queries per second from bots. 15 RPS is very far from an apocalypse.

It is if you're hosting an internet search engine on a PC.

Why would you do such a thing in the first place?

Re: Botspam apocalypse

#229
post #8

This kind of botspam is usually pretty easy to address with redbean using the finger https://redbean.dev/#finger and maxmind https://redbean.dev/#maxmind modules. The approach I usually recommend people isn't so much ip reputation, which can be unfair, but rather it allows you to find evidence of clients lying to you. For example, if the User-Agent says it's Windows with a language preference of English, but the TCP…

[deleted]

Re: Botspam apocalypse

#230

Earlier quoted context omitted.

It is if you're hosting an internet search engine on a PC.

Why would you do such a thing in the first place?

Because I want this search engine to exist, and I'm not a multimillionaire so I can't afford better hardware.

See, when it comes to not being able to find stuff on Google, you can either complain about it on the internet, or you can build a search engine yourself that allows you to find what you are looking for.

I chose the second option.

Post reply on HN