Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ban me at the IP level if you don't like me

boston.conman.org

Post reply on HN
Print view View original

401–410 of 516 posts

Re: Ban me at the IP level if you don't like me

#401

Post by hinkley »

xp84 wrote: post #395
hinkley wrote: post #390

Earlier quoted context omitted.

One of our customers was paying a third party to hit our website with garbage traffic a couple times a week to make sure we were rejecting malformed requests. I was forever tripping over these in Splunk while trying to look for legitimate problems. We also had a period where we generated bad URLs for a week or two, and the worst part was I think they were on links marked nofollow. Three years later there was a bot st…

> they were on links marked nofollow if i'm understanding you correctly you had an indexable page that contained links with nofollow attribute on the tags. It's possible some other mechanism got those URLs into the crawler like a person visiting them? Nofollow on the link won't prevent the URL from being crawled or indexed. If you're returning a 404 for them, you ought to be able to use webmaster tools or whatever it…

The dumbest part is that we’d known about this for a long time and one day someone discovered we’d implemented a feature toggle to remove those URLs and then it just never got turned on, despite being announced that it had.

They were meant to be interactive URLs on search pages. Someone implemented them I think trying to allow A11y to work but the bots were slamming us. We also weren’t doing canonical URLs right in the destination page so they got searched again every scan cycle. So at least three dumb things were going on, but the sorts of mistakes that normal people could make.

Re: Ban me at the IP level if you don't like me

#402

Post by xp84 »

Arnavion wrote: post #294
immibis wrote: post #283

Earlier quoted context omitted.

Let's suppose I'm running a residential proxy. Of course my home IP address changes every day, so you'll end up blocking my entire ISP (a major one) or city (a major one) one by one. And what if I'm behind CGNAT? You will block my entire ISP or city all in one go, and get complaints from a lot of people.

If enough websites block the entire ISP / city in this way, *and* enough users get annoyed by being blocked and switch ISPs, then the ISPs will be motivated to stay in business and police their customers' traffic harder. Alas, the "enough users get annoyed by being blocked and switch ISPs" step will never happen. Most users only care about the big web properties, and those have the resources to absorb such crawler tr…

> the ISPs will be motivated to stay in business and police their customers' traffic harder.

You can be completely forgiven if you're speaking from a non-US perspective, but this made me laugh pretty hard -- in this country we usually have a maximum of one broadband ISP available from any one address.

A small fraction of a few of the most populous, mostly East-coast, cities, have fiber and a highly asymmetrical DOCSIS cable option. The rest of the country generally has the cable option (if suburban or higher density) and possibly a complete joke of ADSL (like 6-12Mbps down).

There is nearly zero competition, most customers can choose to either keep their current ISP or switch to something with far worse speed/bandwidth caps/latency, such as cellular internet, or satellite.

Re: Ban me at the IP level if you don't like me

#403

Post by ikiris »

Bender wrote: post #269
ralferoo wrote: post #264

Earlier quoted context omitted.

> It depends on people knowing about it, and adding the complexity of checking it to their crawler. I can't believe any bot writer doesn't know about robots.txt. They're just so self-obsessed and can't comprehend why the rules should apply to them, because obviously their project is special and it's just everyone else's bot that causes trouble.

(malicious) Bot writers have exactly zero concern for robots.txt. Most bots are malicious. Most bots don't set most of the TCP/IP flags. Their only concern is speed. I block about 99% of port scanning bots by simply dropping any TCP SYN packet that is missing MSS or uses a strange value. The most popular port scanning tool is masscan which does not set MSS and some of the malicious user-agents also set some odd MSS v…

You do realize vpns and older connectivity exists that needs values lower than 1280 right?

Re: Ban me at the IP level if you don't like me

#404

Post by zelphirkalt »

BoredPositron wrote: post #211
zelphirkalt wrote: post #206

Earlier quoted context omitted.

Which will ensure you never get customers from those countries. And so the circle closes ...

The regulatory burden of conducting business with countries like Russia or China is a critical factor that offhand comments like yours consistently overlook.

It is funny how people immediately jump to conclusions, while I was merely pointing out an circular argument. People immediately think I am jumping to "aid one side". Shows much more about them than me, actually.

Re: Ban me at the IP level if you don't like me

#405

Post by JumpCrisscross »

zipliner wrote: post #331
JumpCrisscross wrote: post #125

Earlier quoted context omitted.

> Allow/deny list is more descriptive It really isn’t. It’s a novel term, which implies a functional difference from the common term. Like, I can run around insisting on calling soup food drink because it’s technically more descriptive, that doesn’t mean I’m communicating better. To the extent we have a bug in our language, it’s probably in describing dark brown skin tones as black. Not a problem with the word black…

> It really isn’t. What do the lists do? They allow or deny access, right? Seems allow/deny are fitting descriptive terms for them then. White/black are much more ambiguous prefix terms and and also come with much more semantic baggage. All in all an easy, clarifying change.

> What do the lists do? They allow or deny access, right?

In part. A whitelisted party is always allowed access. If you are whitelisted to enter my home, you always have access. This is different from conditionally having access, or having access for a pre-set period of time.

Same for a blacklist. An IP on a blacklist clearly communicates that it should not be casually overridden in a way a ‘deny-access list’ does not.

> White/black are much more ambiguous prefix terms and and also come with much more semantic baggage

That baggage includes the broadly-understood meaning of the word. When someone says to whitelist an IP address, it’s unambiguous. If someone says to add an IP address to an allow access list, that’s longer and less clear. Inventing a personal language can be an effective way to think through a problem. But it isn’t a way to communicate.

Black and white are colours. (Practically.) I am sympathetic to where folks arguing for this come from. But we aren’t going to solve racism by literally removing black and white from our language.

Re: Ban me at the IP level if you don't like me

#406

Post by kldg »

What is the commonality between websites severely affected by bots? I run web server from home for years on .com TLD, is high-ish in Google site index for relevant keywords, and do not have any exotic protections against bots either on router or server (though I did make an attempt at counting bots, for curiosity). I get very frequent port scans, and they usually grab the index page, but only rarely follow dynamically-loaded links. I don't even really think about bots because there is no noticeable impact either when I ran server on Apache 2, and now with multiple websites run using Axum.

I would guess directory listing? -But I'm an idiot, so any elucidation would be appreciated.

Re: Ban me at the IP level if you don't like me

#407

Post by Dylan16807 »

closewith wrote: post #216
antonkochubey wrote: post #212

Earlier quoted context omitted.

One of requirements of Visa/Mastercard is for the customer to be able to contact merchant post-purchase.

Only via the original method of commerce. An online retailer who geoblocks users does not have to open the geoblock for users who move into the geoblocked regions. I have first-hand experience, as I ran a company that geoblocked US users for legal reasons and successfully defended chargebacks by users who made transactions in the EU and disputed them from the US. Chargebacks outside the US are a true arbitration proc…

"Visiting the website" is the method. It's nonsense to say that visiting from a different location is a different method. I don't care if you won those disputes, you did a bad thing and screwed over your customers.

Re: Ban me at the IP level if you don't like me

#408

Post by jeltz »

fsckboy wrote: post #381
8organicbits wrote: post #235

I've been working on a web crawler and have been trying to make it as friendly as possible. Strictly checking robots.txt, crawling slowly, clear identification in the User Agent string, single IP source address. But I've noticed some anti-bot tricks getting applied to the robot.txt file itself. The latest was a slow loris approach where it takes forever for robots.txt to download. I accidentally treated this as a 404…

> a slow loris approach does this refer to the word loris recently and only after several years being added to Wordle™?

No, why would it? The attack was named after the animal, slow loris, many years ago.

Re: Ban me at the IP level if you don't like me

#409

Post by sidewndr46 »

hinkley wrote: post #390
sidewndr46 wrote: post #324

Earlier quoted context omitted.

I don't think you have any idea how serious the issue is. I was loosely speaking in charge of application-level performance at one job for a web app. I was asked to make the backend as fast as possible at dumping the last byte of HTML back to the user. The problem I ran into was performance was bimodal. We had this one group of users that was lightning fast and the rest were far slower. I chased down a few obvious ou…

One of our customers was paying a third party to hit our website with garbage traffic a couple times a week to make sure we were rejecting malformed requests. I was forever tripping over these in Splunk while trying to look for legitimate problems. We also had a period where we generated bad URLs for a week or two, and the worst part was I think they were on links marked nofollow. Three years later there was a bot st…

I guess my position it was comparatively well behaved? There were bots that would full speed blitz the website, for absolutely no reason. You just scraped this page 27 seconds ago, do you really need to check it for an update again? Also it hasn't had a new post in the past 3 years, is it really going to start being lively again?

Re: Ban me at the IP level if you don't like me

#410

Post by Dylan16807 »

BoredPositron wrote: post #186
ruszki wrote: post #139

Earlier quoted context omitted.

Okay, but this causes me about 90% of my major annoyances. Seriously. It’s almost always these stupid country restrictions. I was in UK. I wanted to buy a movie ticket there. Fuck me, because I have an Austrian ip address, because modern mobile backends pass your traffic through your home mobile operator. So I tried to use a VPN. Fuck me, VPN endpoints are blocked also. I wanted to buy a Belgian train ticket still fr…

The vpn is probably your problem there mate.

They tried a VPN as a backup for one of those problems.

So no. It's not.

Post reply on HN