Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Botspam apocalypse

memex.marginalia.nu

Post reply on HN
Print view View original

231–240 of 358 posts

Re: Botspam apocalypse

#231

Post by marginalia_nu »

b112 wrote: post #219
dspillett wrote: post #216

Earlier quoted context omitted.

If doing something fends off a lot of bots, but also inconveniences a very small number of people who have significantly non-standard or just out-of-date configurations, I'm likely to favour protecting myself from the former over worrying about the latter. To paraphrase Mr Spok: The inconveniences of the me outweigh the inconveniences of the you!

Bear in mind, inconveniencing 4.8% of users, does not map identically. Instead, you are often dumping 4.8+4.8+4.8 as you add block methods, with some overlap.

To be fair, most of my visitors are not exactly lining up with the expectations of "standard". I get >90% of my [human] traffic from desktop clients, for example.

Re: Botspam apocalypse

#232

Post by jimmywetnips »

baisq wrote: post #128
nixcraft wrote: post #95

I run a popular blog and confirm that spam is a massive issue. I am trying to keep the independent web alive with an old-school commenting system because it helps readers and myself improve outdated posts. My domain is over 20+ years old and attracts all sorts of threats, including monthly DDoS and daily spam. Using Cloudflare solved all of these problems. Next, you need to add firewall rules inside Cloudflare WAF to…

People here like to say that BigCo has ruined the independent web and that everything is now siloed and blablabla but the truth is that running an independent website fucking sucks in many regards

Exactly, sorry the current solutions don't live up to ideals, but if there's no better then we're already doing the ideal thing

Re: Botspam apocalypse

#233

Post by RamRodification »

marginalia_nu wrote: post #190
ajsnigrutin wrote: post #173

Earlier quoted context omitted.

> - Rate-limit everything, absolutely everything. Set sane limits. This breaks when multiple users are behind the same IP. I've seen services fail even in classroom, because the prof did something and a few tens of students followed (captchas everywhere).

(Author) I do in fact rate-limit everything, it is good advice, but the way you implement rate-limiting allows for traffic bursts. It's basically a reverse leaky bucket, where you start out with N allowed requests, which gets depleted for each request, and refilled slowly over time. Search traffic is fairly bursty, people do a few requests where they tweak the query and then they go go away.

Off-topic, but isn't that a normal (non-reverse) leaky bucket? When the bucket gets full the rate limiting engages. An empty bucket allows for a burst without getting full. It slowly leaks over time at a rate that allows a normal amount of traffic without filling up.

Re: Botspam apocalypse

#234

Post by marginalia_nu »

RamRodification wrote: post #233
marginalia_nu wrote: post #190

Earlier quoted context omitted.

(Author) I do in fact rate-limit everything, it is good advice, but the way you implement rate-limiting allows for traffic bursts. It's basically a reverse leaky bucket, where you start out with N allowed requests, which gets depleted for each request, and refilled slowly over time. Search traffic is fairly bursty, people do a few requests where they tweak the query and then they go go away.

Off-topic, but isn't that a normal (non-reverse) leaky bucket? When the bucket gets full the rate limiting engages. An empty bucket allows for a burst without getting full. It slowly leaks over time at a rate that allows a normal amount of traffic without filling up.

Hmm, yeah, that's actually true now that I think about it.

Re: Botspam apocalypse

#235

Post by js4ever »

"There has been upwards of 15 queries per second from bots. There is just no way to deal with that sort of traffic, barely even to reject it."

What??? My phone can serve that easily, any modern server can handle 50-250 rps

Re: Botspam apocalypse

#236

Post by marginalia_nu »

efitz wrote: post #41
BiteCode_dev wrote: post #31

It's not that bad. First, of course, you have cloudflare and recaptcha, which are free and very efficient, as the author say. But even if you don't want to use them (some of my services don't), most bots are very dumb: - require JS, and you lose half of the web ones - silly tricks like hidden input fields in forms that worked in 2000 still work in 2022. Use a bunch of them, and you can yet again halve the bot traffic…

Also, attackers are rarely going to try to guess your URLs - they’re going to find them via Google or Shodan, or, if you’re a good rest citizen, via “/ /“

I get quite a lot of guesses in my logs, probing in /solr/ and so on (fairly pointlessly, I might add, as I run bespoke software).

Re: Botspam apocalypse

#237

Post by marginalia_nu »

Aachen wrote: post #177

> There has been upwards of 15 queries per second from bots. There is just no way to deal with that sort of traffic, barely even to reject it. If the queries are not a megabit each, you're doing way too much processing before applying rate limiting. Rejecting traffic ought not to take more than 1-2 milliseconds, even if you need to look up an api key or IP address in the database. I, too, host services on a residenti…

That's queries per second (as searches), not requests per second.

You know how blogs sometimes can't handle being on the hacker news front page from all the traffic? Well my search engine has survived that. That was 1-2 QPS.

Unmitigated bot-traffic is roughly 10x the traffic of a hacker news death hug, as a sustained load.

Re: Botspam apocalypse

#238

Post by julianlam »

I disagree with TFA's take on dealing with spam — giving up!

For our app, we don't deal with spam in any novel way. We use honey pot, SFS, and Akismet.

However, by far the easiest way to stop spammers is a post queue. Lots of spammers will just create a burner account, fire off their spam, and start over. Given no actual reputation, give them the trust they deserve — none.

The other factor is building out a fast backend. Besides benefiting your own users, it also means Googlebot or Ahrefsbot won't absolutely cripple your site when they come knocking. Sometimes that is doable, sometimes not.

Re: Botspam apocalypse

#239

Post by marginalia_nu »

julianlam wrote: post #238

I disagree with TFA's take on dealing with spam — giving up! For our app, we don't deal with spam in any novel way. We use honey pot, SFS, and Akismet. However, by far the easiest way to stop spammers is a post queue. Lots of spammers will just create a burner account, fire off their spam, and start over. Given no actual reputation, give them the trust they deserve — none. The other factor is building out a fast back…

(Author) I'm running a search engine though. Do you propose I require users to register an account, and then not allow them to search?

I think my backend is plenty fast given it's hosted on a PC off domestic broadband. Most searches complete sub-100ms.

Re: Botspam apocalypse

#240

Post by NullPrefix »

Avamander wrote: post #156
gkbrk wrote: post #147

Earlier quoted context omitted.

> If you can, require HTTP/2.0. Bots break. Non-bots break as well. I have Firefox configured to use HTTP/1.1 only. No reason to chase Google's standard-of-the-day, HTTP/1.1 has worked for ages and it will continue to do so for the foreseeable future.

Some old browsers break as well, if it's worth it depends on the website. It's your prerogative to disable an useful feature, you can also disable JavaScript. But there's little reason for a website operator to cater to that unnecessary edge case if it's mostly used for abuse.

JavaScript is mostly used for abuse.
Post reply on HN