Botspam apocalypse • Hacker News Board

Re: Botspam apocalypse

#291

Post by LinuxBender » Thu, Aug 04, 2022, 4:06 PM UTC

CodeSgt wrote: post #222

Avamander wrote: post #108

It's annoying for sure. I deal with abuse at a large scale. I'd recommend: - Rate-limit everything, absolutely everything. Set sane limits. - Rate-limit POST requests harder. Preferably dynamically based on geoip. - Rate-limit login and comment POST requests even harder. Ban IPs that exceed the amount. - Require TLS. Drop TLSv1.0 and TLSv1.1. Bots certainly break. - Require SNI. Do not reply without SNI (nginx has 44…

> Rate-limit login and comment POST requests even harder. Ban IPs that exceed the amount Don't ban IPs. Or if you do, let the ban expire relatively quickly (days/weeks, not months/years).

Or at least rate limit session cookies. If a person does not have a session cookie, rate limit by IP. If they are authenticated as a unique person have different rate limits and different levels of authentication. HAProxy can do different rate limits by ACL conditions.

Or instead of strictly rate limiting, ask them a question that can't be "looked up" in a table and that requires human thought, philosophy, emotion, ethics. Maybe GPT could eventually adapt to this and in that case fall back to IP rate limiting and grow the set of questions.

Re: Botspam apocalypse

#292

Post by Test0129 » Thu, Aug 04, 2022, 4:17 PM UTC

Not totally unrelated but I had to turn off email alerts and come up with a way to summarize things because Fail2Ban and other alert systems were hit quite literally every 15 seconds with port scans/attempted entries on SSH and other ports. Reporting the abuse to ARIN/ICANN didn't help because almost a full 95% of the traffic originated from China, and 90% of the remaining 5% was Russia. Of the remaining they were zombies inside of America, typically on digital ocean, and I was able to get those handled quickly and efficiently. When I had a simple (secure) login system hosted on HTTPS it was getting hit hard enough my VPS ISP was sending emails to figure out a way to stop it. There are literally 3 people that even know of the existence of these services.

It is actually nuts just how much bot spam there is.

Re: Botspam apocalypse

#293

Post by Avamander » Thu, Aug 04, 2022, 4:18 PM UTC

jmt_ wrote: post #290

Avamander wrote: post #108

It's annoying for sure. I deal with abuse at a large scale. I'd recommend: - Rate-limit everything, absolutely everything. Set sane limits. - Rate-limit POST requests harder. Preferably dynamically based on geoip. - Rate-limit login and comment POST requests even harder. Ban IPs that exceed the amount. - Require TLS. Drop TLSv1.0 and TLSv1.1. Bots certainly break. - Require SNI. Do not reply without SNI (nginx has 44…

I'm not very familiar with all the workings of HTTP/2.0 - why would it break bots? Assuming no CloudFlare type protection, does it somehow stop someone from using curl to get (non-JS generated) content? Does it thwart someone accessing the site from something like playwright/selenium?

> I'm not very familiar with all the workings of HTTP/2.0 - why would it break bots?

There's a lot of outdated garbage bots out there. Not using HTTP/2.0 is also often the default with various HTTP libraries.

Re: Botspam apocalypse

#294

Post by [deleted] » Thu, Aug 04, 2022, 4:19 PM UTC

OliverJones wrote: post #164

kazinator wrote: post #15

In the 1980's, we kept anklebyters off dial-up BBSses with a simple technique: voice validation. To join the forum, you had to fill an application first, which included your real name and phone number. The sysop would give you a call for a quick chat, and then grant you access if you didn't seem like a twit. This would be entirely practical for some small-time operator trying to run a forum off residential broadband,…

"anklebyters"! I learned a useful new word today. Thanks.

[deleted]

Re: Botspam apocalypse

#295

Post by fariszr » Thu, Aug 04, 2022, 4:22 PM UTC

nixcraft wrote: post #95

I run a popular blog and confirm that spam is a massive issue. I am trying to keep the independent web alive with an old-school commenting system because it helps readers and myself improve outdated posts. My domain is over 20+ years old and attracts all sorts of threats, including monthly DDoS and daily spam. Using Cloudflare solved all of these problems. Next, you need to add firewall rules inside Cloudflare WAF to…

Hmmm, maybe offload your comments to something else ?

Im thinking of using GitHub issues/discussions as a comment system. The website and everything will function normally without CloudFlare, but the comments are based on GitHub, which will deal with spam and hosting for me.

And i personally think using it is better from centralizing the internet view, as you don't increase the absolutely crazy 20% that CF controls of the web.

But that obviously doesn't solve the DDOS problem, which should be solved with big cloud providers which have included DDOS protection.

Another workaround is using IPFS, but for a normal user, he/she will need a gateway, and guess who operates on of the biggest IPFS gateways ?, Yes CF. That without considering the tradeoffs is using IPFS.

I think a static site + external comment provider like Github, might help with the attacks and spam without using Cloudflare, but I don't have any website close to your blogs size, so its all just a predication.

Re: Botspam apocalypse

#296

Post by CWuestefeld » Thu, Aug 04, 2022, 4:25 PM UTC

ajsnigrutin wrote: post #173

Avamander wrote: post #108

It's annoying for sure. I deal with abuse at a large scale. I'd recommend: - Rate-limit everything, absolutely everything. Set sane limits. - Rate-limit POST requests harder. Preferably dynamically based on geoip. - Rate-limit login and comment POST requests even harder. Ban IPs that exceed the amount. - Require TLS. Drop TLSv1.0 and TLSv1.1. Bots certainly break. - Require SNI. Do not reply without SNI (nginx has 44…

> - Rate-limit everything, absolutely everything. Set sane limits. This breaks when multiple users are behind the same IP. I've seen services fail even in classroom, because the prof did something and a few tens of students followed (captchas everywhere).

For sure! Our site is B2B ecommerce, and any sizeable customer has all their users coming to us from a single NAT or proxy. For major customers it's likely that there are several of their employees using our system at any given time.

The answer needs a whole lot more finesse than this.

Re: Botspam apocalypse

#297

Post by seydor » Thu, Aug 04, 2022, 4:25 PM UTC

paulmd wrote: post #2

> They're a major part in killing off web forums, and a significant wet blanket on any sort of fun internet creativity or experimentation. > The only ones that can survive the robot apocalypse is large web services. Your reddits, and facebooks, and twitters, and SaaS-comment fields, and discords. They have the economies of scale to develop viable countermeasures, to hire teams of people to work on the problem full ti…

Alternatively, allow $0 signups but approve every new account. It's rather easy to spot spam signups

Re: Botspam apocalypse

#298

Post by CWuestefeld » Thu, Aug 04, 2022, 4:26 PM UTC

shiftpgdn wrote: post #283

Avamander wrote: post #186

Earlier quoted context omitted.

Yep, there isn't a silver bullet that curtails all abuse.

Blocking the entire aws/gcp/azure/digital ocean/linode IP ranges will stop 99.999% of malicious bot traffic full stop.

Yes, it would.

It would also stop a not-insignificant number of my customers.

Re: Botspam apocalypse

#299

Post by seydor » Thu, Aug 04, 2022, 4:27 PM UTC

nixcraft wrote: post #95

I run a popular blog and confirm that spam is a massive issue. I am trying to keep the independent web alive with an old-school commenting system because it helps readers and myself improve outdated posts. My domain is over 20+ years old and attracts all sorts of threats, including monthly DDoS and daily spam. Using Cloudflare solved all of these problems. Next, you need to add firewall rules inside Cloudflare WAF to…

Unfortunately, well-known platforms with known URIs are targetted way more than any custom website. I think if wordpress just allowed rewriting all the URLs would reduce spam attacks by a lot

Re: Botspam apocalypse

#300

Post by megous » Thu, Aug 04, 2022, 4:37 PM UTC

RamRodification wrote: post #233

marginalia_nu wrote: post #190

Earlier quoted context omitted.

(Author) I do in fact rate-limit everything, it is good advice, but the way you implement rate-limiting allows for traffic bursts. It's basically a reverse leaky bucket, where you start out with N allowed requests, which gets depleted for each request, and refilled slowly over time. Search traffic is fairly bursty, people do a few requests where they tweak the query and then they go go away.

Off-topic, but isn't that a normal (non-reverse) leaky bucket? When the bucket gets full the rate limiting engages. An empty bucket allows for a burst without getting full. It slowly leaks over time at a rate that allows a normal amount of traffic without filling up.

To me, it's a bucket that's being filled at constant rate from a tap until it's full, and the traffic requires taking some water from the bucket. If there's no water, the trafic has to be dropped or wait in a queue.

Basicaly, you can look at it either way.