Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ban me at the IP level if you don't like me

boston.conman.org

Post reply on HN
Print view View original

421–430 of 516 posts

Re: Ban me at the IP level if you don't like me

#421

Post by cyanydeez »

brianwawok wrote: post #301
8organicbits wrote: post #235

I've been working on a web crawler and have been trying to make it as friendly as possible. Strictly checking robots.txt, crawling slowly, clear identification in the User Agent string, single IP source address. But I've noticed some anti-bot tricks getting applied to the robot.txt file itself. The latest was a slow loris approach where it takes forever for robots.txt to download. I accidentally treated this as a 404…

I doubt that’s on purpose. The bad guys that don’t follow robots don’t bother downloading it. Never assume malice what can be attributed to incompetence.

Its likely just a shitty attempt to rate limit bots

Re: Ban me at the IP level if you don't like me

#422

Post by simoncion »

TZubiri wrote: post #417
socalgal2 wrote: post #415

Aren't many apartment buildings all coming from just a few IP addresses? https://en.wikipedia.org/wiki/Carrier-grade_NAT

Yes, and this makes ip banning have false positives. But ultimately it's worth it, you are responsible for your neighbours.

> [Y]ou are responsible for [how] your neighbours [use the Internet].

Nope.

I'm very much not responsible for snooping on my neighbor's private communications. If anyone is responsible for doing any sort of abuse monitoring, it is the ISP chosen by my neighbor.

Re: Ban me at the IP level if you don't like me

#423

Post by Mo3 »

imiric wrote: post #261
seer wrote: post #214

Earlier quoted context omitted.

I guess this is what "Identity aware proxy" from GCP can do for you? Outsource all of this to google - where you can connect your own identity servers, and then your service will only be accessed after the identity has been verified. We have been using that instead of VPN and it has been incredibly nice and performant.

Yeah, I suppose it's something like that. Except that my solution wouldn't rely on Google, would be open source and self-hostable. Are you aware of a similar project that does this? Would save me some time and effort. :) There also might be similar solutions for other cloud providers or some Kubernetes-adjacent abomination, but I specifically want something generic and standalone.

https://github.com/topics/identity-aware-proxy

Re: Ban me at the IP level if you don't like me

#424

Post by gucci-on-fleek »

kldg wrote: post #406

What is the commonality between websites severely affected by bots? I run web server from home for years on .com TLD, is high-ish in Google site index for relevant keywords, and do not have any exotic protections against bots either on router or server (though I did make an attempt at counting bots, for curiosity). I get very frequent port scans, and they usually grab the index page, but only rarely follow dynamicall…

For my personal site, I let the bots do whatever they want—it's a static site with like 12 pages, so they'd essentially need to saturate the (gigabit) network before causing me any problems.

On the other hand, I had to deploy Anubis for the SVN web interface for tug.org. SVN is way slower than Git (most pages take 5 seconds to load), and the server didn't even have basic caching enabled, but before last year, there weren't any issues. But starting early this year, the bots started scraping every revision, and since the repo is 20+ years old and has 300k files, there are a lot of pages to scrape. This was overloading the entire server, making every other service hosted there unusable. I tried adding caching and blocking some bad ASNs, but Anubis was (unfortunately) the only solution that seems to have worked.

So, I think that the main commonality is popular-ish sites with lots of pages that are computationally-expensive to generate.

Re: Ban me at the IP level if you don't like me

#425

Post by mjmas »

Sohcahtoa82 wrote: post #338
sidewndr46 wrote: post #327

Earlier quoted context omitted.

It's even funnier when you realize it is a request for a known exploit in WordPress. Does someone really run that on port 22?

I HAVE heard of someone that runs SSH on port 443 and HTTPS on 22. It blocks a lot of bots, but I feel like just running on a high port number (10,000+) would likely do better.

I have a service running on a high port number on just a straight IPv4 and it does get a bit of bot traffic, but they are generally easy to filter out when looking at logs (well behaved ones have a domain in their User-Agent and bingbot takes my robots.txt into account. I dont think I've seen the Google crawler. Other bots can generally be worked out as anything that didn't request my manifest.json a few seconds after loading the main page)

Re: Ban me at the IP level if you don't like me

#427

Post by dilyevsky »

hinkley wrote: post #390
sidewndr46 wrote: post #324

Earlier quoted context omitted.

I don't think you have any idea how serious the issue is. I was loosely speaking in charge of application-level performance at one job for a web app. I was asked to make the backend as fast as possible at dumping the last byte of HTML back to the user. The problem I ran into was performance was bimodal. We had this one group of users that was lightning fast and the rest were far slower. I chased down a few obvious ou…

One of our customers was paying a third party to hit our website with garbage traffic a couple times a week to make sure we were rejecting malformed requests. I was forever tripping over these in Splunk while trying to look for legitimate problems. We also had a period where we generated bad URLs for a week or two, and the worst part was I think they were on links marked nofollow. Three years later there was a bot st…

> And if you 429 Google’s bots they will reduce your pagerank. That’s straight up extortion from a company that also sells cloud services.

Googlebot uses different IP space from gcp

Re: Ban me at the IP level if you don't like me

#428

Post by pabs3 »

johnisgood wrote: post #154
pabs3 wrote: post #152

Earlier quoted context omitted.

Which site is it?

My own shitty personal website that is so uninteresting that I do not even wish to disclose here. Hence my lack of understanding of the down-votes for me doing what works for my OWN shitty website, well, server. In fact, I bet it would choke on a small amount of traffic from here considering it has a shitty vCPU with 512 MB RAM.

Personal sites are definitely interesting, way more interesting than most of the rest of the web.

I was thinking I would put your site into archive.org, using ArchiveBot, with reasonable crawl delay, so that it is preserved if your hardware dies. Ask on the ArchiveTeam IRC if you want that to happen.

https://chat.hackint.org/?join=%23archiveteam-bs

Re: Ban me at the IP level if you don't like me

#429

Post by dang »

JohnMakin wrote: post #391
dang wrote: post #389

Earlier quoted context omitted.

Please don't cross into personal attack. You can make your substantive points without that.

I disagree with that characterization of the post - merely commenting that a user that could come away with this take has never managed a web-facing service, because you'd immediately see the traffic is immense and constant, especially from crawlers. Sorry if I didn't elaborate that point clearly enough, point taken, I will more carefully craft such responses so such a point isn't misinterpreted or flagged.

I'm sure that would help, yes. Also, there's no need to phrase such a comment in terms of someone else lacking any experience of X - there are too many ways to get that wrong, and even if you're right, it can easily come across as a putdown. If you'd made your point in this case, for example, in terms of your own experience managing a web-facing service, you could have included all the same useful information, if not more!

(One other thing is that the "tell me without telling me" thing is an internet trope and the site guidelines ask people to avoid those - they tend to make for unsubstantive comments, plus they're repetitive and we're trying to avoid that here. But I just mention this for completeness - it's secondary to the other point.)

Re: Ban me at the IP level if you don't like me

#430

Post by dang »

phyzome wrote: post #394
dang wrote: post #389

Earlier quoted context omitted.

Please don't cross into personal attack. You can make your substantive points without that.

It think it's fair play to claim that someone doesn't have relevant experience when it seems very clear that they do not.

It's too easy for these things to seem clear and then turn out not to be right at all; moreover there's no need to get personal about these things - it has no benefit and there's an obvious cost.
Post reply on HN