Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Botspam apocalypse

memex.marginalia.nu

Post reply on HN
Print view View original

101–110 of 358 posts

Re: Botspam apocalypse

#101

Post by jaclaz »

probably_wrong wrote: post #35
hamilyon2 wrote: post #5

I experienced this firsthand with government immigration websites. The thing is there are only so many time slots and and people are forsed to use a certain web site to apply, so everyone is hunting for available time and generally none are available. So, some creative people set up bots which check periodically for them. They are paid services which will do that for you. Now we have bots hammering gatekeeper's websi…

> I don't think there is a good solution besides a deep anti-bot expertise whithin the primary website development team But there is a solution: the website team should get their act together and remove the "first come first served" aspect altogether. Do you, citizen, want to register? Cool - leave your e-mail and we'll call you. Is the service optional? Then we'll pick at random from the pool of applicants and e-mai…

>Do you, citizen, want to register? Cool - leave your e-mail and we'll call you. Is the service optional? Then we'll pick at random from the pool of applicants and e-mail them. Is the service mandatory? Then sign up and we'll call you once you reach the top of the queue.

Wait a minute, wouldn't that be more work for "us"?

Let me think ....

Re: Botspam apocalypse

#102

Post by Joel_Mckay »

philprx wrote: post #68
Joel_Mckay wrote: post #36

For small sites, I would just use a simple firewall: 1. whitelist the finite IP ranges for the regional ISPs/country where you do business 2. blacklist the proxy and tor exit nodes 3. blacklist the list of published compromised servers 4. add spamhaus blacklists 5. add fail2ban rules to trip on common server security scans, and unused common service ports 6. publicly reply to those having access issues, and imply the…

Tor users are often legitimate good internet citizens. A lot of (lucky) us have the luxury to live in real democracies. Some others live in countries that use every single aspect of their private lives (DPI, mass surveillance) to put pressure on them and bend them to the regime's will. In my opinion, Tor and anonymity should not be killed as a result of silly bots.

Your opinion is duly noted, and I agree most knowledge should be equally accessible to give everyone a chance to grow.

That being said, a commercial site owes nothing to financially irrelevant bandits, sociopaths, or shills.

Try it for a week, and then weigh the liability again. ;)

Re: Botspam apocalypse

#103

Post by mobilio »

EdwardDiego wrote: post #88
sparkling wrote: post #47

Earlier quoted context omitted.

Very nice list of countermeasures. I agree that doing these small things like hidden input fields really go a long way. I would add to that: - block signups/comments from known throwaway email domains - block known datacenter IP ranges, at least for POST requests. Honestly on our sites 50% of spam was coming from AWS EC2 IPs - use a proxy/vpn/bot detection service like https://focsec.com

Yup, in adtech, "IP is an AWS block" was a bot 99.999% of the time. The 0.001% was that person using EC2 as a proxy or VPN server.

It's not only AWS. Also happens on Azure and GCP.

Re: Botspam apocalypse

#104

Post by cmjs »

tmikaeld wrote: post #23
timmaxw wrote: post #21

I wonder if proof-of-work would help. Suppose every form submission requires an expensive calculation, calibrated to take about 1 second on a typical modern computer/smartphone. For human users, this happens in the background, although it makes the website feel slower. But for bots, it dramatically limits how many submissions each botnet host can make to random websites.

"mCaptcha uses SHA256 based proof-of-work(PoW) to rate limit users." https://github.com/mCaptcha/mCaptcha

I'm curious whether this can actually be considered to be a "CAPTCHA" in the true sense of the term. It doesn't seem to be intended to "tell computers and humans apart", but rather to force the client computer (not the human user) to do some work in order to slow down DOS attacks.

Of course slowing down DOS attacks is a great goal in itself, and it's very often what captchas have been (ab)used for, but it doesn't seem to me to replace all or most use cases for a captcha. In particular, since it can be completed by an automated system at least as easily as by a human, it doesn't seem like it would limit spambot signups or spambot comment or contact form submissions in any meaningful way.

Or am I misunderstanding, @realaravinth?

Re: Botspam apocalypse

#105

Post by ori_b »

closewith wrote: post #82
paulmd wrote: post #60

Earlier quoted context omitted.

few edits I wanted to make but couldn't while HN was down: this comes to a question of intentions, right? Like are you trying to build a high-value community , or are you trying to make a billion-dollar company? Photrio or Pentaxforums is never going to sell for a billion dollars like Reddit, and that's not the kind of community that Reddit is trying to build. The highly-chaotic multithreaded model of Reddit/HN/etc i…

> The highly-chaotic multithreaded model of Reddit/HN/etc is directly designed to be impenetrable and chaotic, where everyone is just responding to everyone rather than having a "flow of conversation" in which everyone is involved. I couldn’t disagree more with this characterisation. Part of the reason that sites like Reddit and HN are preferred to traditional fora (which have their own engagement mechanisms) is beca…

It's something borrowed directly from email, at least in traditional clients, and it works.

Re: Botspam apocalypse

#106

Post by [deleted] »

golergka wrote: post #84

> If Marginalia Search didn't use Cloudflare, it couldn't serve traffic. There has been upwards of 15 queries per second from bots. 15 RPS is very far from an apocalypse.

[deleted]

Re: Botspam apocalypse

#107

Post by Macha »

closewith wrote: post #82
paulmd wrote: post #60

Earlier quoted context omitted.

few edits I wanted to make but couldn't while HN was down: this comes to a question of intentions, right? Like are you trying to build a high-value community , or are you trying to make a billion-dollar company? Photrio or Pentaxforums is never going to sell for a billion dollars like Reddit, and that's not the kind of community that Reddit is trying to build. The highly-chaotic multithreaded model of Reddit/HN/etc i…

> The highly-chaotic multithreaded model of Reddit/HN/etc is directly designed to be impenetrable and chaotic, where everyone is just responding to everyone rather than having a "flow of conversation" in which everyone is involved. I couldn’t disagree more with this characterisation. Part of the reason that sites like Reddit and HN are preferred to traditional fora (which have their own engagement mechanisms) is beca…

This wasn't always through - the first web forums were threaded more often than not, and even vbulletin supported a threaded mode as a user preference well into the 00s (though as it was no longer the default, others were not using linked replies, which hurt its usefulness at that point). This is probably related to imitating the way mailing lists worked, as some of these early UIs were like the older style of mailing list presentation with a few more forms attached.

The flat forums were considered an innovation for a while because "normal users will never understand this nerdy threading model". Arguably to some extent they were right, as mass market products like youtube, facebook, etc. still limit to one level of replies.

The real innovation of the social news sites was the voting and scoring algorithms, which made it manageable by presenting users with the most popular subthreads first, rather than the chronological order the forums had used. And gaining those points had a kind of skinner box effect on keeping users hooked on the sites, which helped their growth too - especially when points used to have a much more prominent display.

Re: Botspam apocalypse

#108

Post by Avamander »

It's annoying for sure. I deal with abuse at a large scale.

I'd recommend:

- Rate-limit everything, absolutely everything. Set sane limits.

- Rate-limit POST requests harder. Preferably dynamically based on geoip.

- Rate-limit login and comment POST requests even harder. Ban IPs that exceed the amount.

- Require TLS. Drop TLSv1.0 and TLSv1.1. Bots certainly break.

- Require SNI. Do not reply without SNI (nginx has 444 return code for that). Ban IP's on first hit that connect without. There's no legitimate use and you'll also disappear from places like Shodan.

- If you can, require HTTP/2.0. Bots break.

- Ban IP's listed on StopForumSpam, ban destination e-mail addresses listed there. If possible also contribute back to SFS and AbuseIPDB.

- Collect JA3 hashes, figure out malicious ones, ban IPs that use those hashes. This blocks a lot of shit trivially because targeting tools instead of behaviour is accurate.

Re: Botspam apocalypse

#109

Post by 12907835202 »

For my forum with 500k users a month I just added a registration captcha related to my niche. E.g. for a Dark Souls forum it would say "what game is this forum about?" And if you got it wrong the validation would include "tip it's just two words Drk S*ls". This reduced spam by over 99% and didn't annoy people with recaptcha.

If someone was unable to get past that captcha (it still happens I have logs!) I figured they were probably not that valuable a contributor anyway.

If someone wanted to target my site directly they could but hasn't happened so far.

Re: Botspam apocalypse

#110

Post by lordgrenville »

TheCapeGreek wrote: post #61
hamilyon2 wrote: post #5

I experienced this firsthand with government immigration websites. The thing is there are only so many time slots and and people are forsed to use a certain web site to apply, so everyone is hunting for available time and generally none are available. So, some creative people set up bots which check periodically for them. They are paid services which will do that for you. Now we have bots hammering gatekeeper's websi…

It gets more fun when there are arbitrary restrictions on process. Fun anecdote: South Africa's Home Affairs website, where you make bookings for passports, only lets you book appointment dates 2 weeks ahead normally. It's effectively permanently booked even without bots this way. Luckily, if you're technically inclined, editing the value of the input element via dev tools is accepted by the form.

It's not filtered out on the back end?
Post reply on HN