41–50 of 158 posts
and all valid traffic too, judging by these HN comments (and my own attempts to connect).
The comments can be misleading. About 2600 people thus far and a few bots have been able to view it. Sunday is the best day to see all the unusual browsers and applications people browse the web with. Weekdays tend to be more mainstream bog standard browsers. This is a good test.
Earlier quoted context omitted.
The comments can be misleading. About 2600 people thus far and a few bots have been able to view it. Sunday is the best day to see all the unusual browsers and applications people browse the web with. Weekdays tend to be more mainstream bog standard browsers. This is a good test.
Rejects me too. Not a bot and connecting directly with a "normal" old-fashioned up-to-date web browser. How will you evaluate how many humans vs bots you blocked?
I just use fail2ban and it seems to do a pretty good job, but you do need to pay attention for the first couple of months, to tune it to your situation. I use one filter that is really just failregex = ^ - \S+ \[\] ".*?" 40[034], and then if that catches something I add it to another more specific list. So far I have about 80 specific regex's and it has stopped all of the traffic. I haven't had anything fallthrough t…
If it is at all helpful most layer-7 load balancers have an option to enable a header that contains the real IP address, then one need only reconfigure their web daemon to log that header. It's very similar to how CDN's send the real IP in a header.
Earlier quoted context omitted.
I would be interested in some examples of valid traffic that they would block. For the purposes of the document I do not consider anything from a data-center to be valid traffic. People can of course skip any or all steps, experiment with one at a time on a test server as they should.
VPN exits often "come from datacenters". And there are entire countries where the web can be unusable without. If you don't have LaLiga in your country, that doesn't mean everyone is so lucky. Blanket IP range bans, in this day and age? Basically a proclamation of incompetence.
I disagree. If I'm getting problematic traffic from even a few of your IPs and they're in a DC/VPS range, I'm blocking it and moving on with my day.
But I suppose none of my clients services typically ever need to be accessible from such countries anyway.
The post content is great. I personally hate the way Cloudfare has been the „default answer“ for the bot problem because Cloudfare has become the most successful MITM attack in history. We need content like this to keep the internet alive. The added explanations by the author in this comment thread are hilarious. You sir are a good writer.
Thankyou for the kind words. I am not for everyone which is the way I would like to keep it. I agree there need to be more alternatives to a big centralized MitM caching proxy even if they can be a bit rough around the edges like mine. There is always room for improvement if enough of us try to come up with alternatives and options.
Exactly! Like any ecosystem, the internet needs variability in order to be healthy and survive long term.
Monocultures are not healthy.
There's a special place in hell for people who block curl and wget, especially on sites with downloadable files (eg source code tgz's, media, etc.), basically anything i might need to wget on a server.
Although @Bender's unique answer is better than my obvious observation.
Earlier quoted context omitted.
I couldn't. Amusing that archive.ph's own crawler, evidently, went through just fine.
I disabled blocking for a few hours then crawled several of my pages with their site.
*I think they do something manually sometimes, like logging into a The Information Pro account and mirroring popular articles. (Unless they can use a visitor’s account somehow, like with some browser extension…) And suspected without evidence maybe The Information did some watermarking to ensure they could quickly ban accounts used like that.
And thank you!
Earlier quoted context omitted.
I disabled blocking for a few hours then crawled several of my pages with their site.
Oh, I assumed Archive had an automated* tool in their arsenal to overcome the blocks that would’ve affected them. Didja happen to confirm they were blocked initially? *I think they do something manually sometimes, like logging into a The Information Pro account and mirroring popular articles. (Unless they can use a visitor’s account somehow, like with some browser extension…) And suspected without evidence maybe The…
They were the reason I started blocking data-centers. I wanted to see if I could block all of their addresses which turned into quite an exercise. Initially I was able to block them by blocking TCP SYN packets with an MSS of 1380 and 1300 but talking about it here was a mistake as their admin is on HN. They fixed that and made their MSS 1460 like a nominal home internet connection.
The next thing I had to block was the Russian Federation. That seems to be where the controlling nodes are, I think. That made me even more curious who they were so I just kept iterating through the blocking process. I ended up blocking a number of ASN's and all of Russia. They are all in the linked archive file in the article but I don't specifically call out the ones that are archive.is.
Most on HN think they operate on a shoe-string budget but I have my doubts. I suspect an asset of a technical arm of an intelligence agency but to what purpose and what end I have no clue. They are very clever whoever they are and certainly earned my respect. Either way I think their site is useful, I just wanted to see if I could block it.