Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

651–660 of 693 posts

Re: Creepy Crawlies

#651

Post by pmlnr »

VladVladikoff wrote: post #357
inigyou wrote: post #301

Earlier quoted context omitted.

So?

Have you tried blocking a million IPs before? Fail2ban gets pretty shaky at even 200,000 The AI crawler traffic I’ve seen sends one request per ip and seemingly has an infinite pool of residential IPs. You can’t block the ASNs becuase you also block honest clients. IP blocks are the wrong solution. And because I’m being negative I’ll also be constructive, IMHO the correct solution for fighting residential proxy crawl…

Fail2ban becomes a serious bottleneck at significant traffic. I've replaced it with a shell script and direct pf commands that run every few minutes.

Re: Creepy Crawlies

#652

Post by roughly »

inigyou wrote: post #519
roughly wrote: post #339

Earlier quoted context omitted.

Wait, so you’ve been the person who wanted to keep people from scraping your site, the person who’s trying to scrape your site, and the person getting paid to help someone scrape your site? Brother, what are you doing with your life?

Welcome to capitalism. Welcome to game theory. Welcome to competition. Welcome to the real world. Welcome to being a grown adult.

Contingency exists, I get that, and if you're truly in that state, sure, I don't judge necessity. A lot of our cohort seems to confuse a studied disinterest in looking beyond the end of their own nose for the sage wisdom of adulthood, though.

Re: Creepy Crawlies

#653

Post by dzhiurgis »

dzhiurgis wrote: post #494

These are most likely not training scrapers, but people looking for concrete pieces of information (i.e. commit, comment, etc).

Clarification - these are crawlers that llm’s use when you ask them to check this site for foo.

Re: Creepy Crawlies

#654

Post by tptacek »

crote wrote: post #577
tptacek wrote: post #145

Earlier quoted context omitted.

I'm not moralizing. The sites running Anubis are owned by people who like Anubis and are fully within their rights to run Anubis. I don't care how enthusiastic they are about it. It's their call. I'm just saying, it's bad computer science.

The real world doesn't care about the soundness of your computer science. All that matters is whether it works. It's like saying we should just completely give up on all NP-hard problems because we'll never discover a way to find the optimal solution in all cases. Meanwhile in the real world, people are more than happy with, say, a route planner which virtually all of the time gives them a solution which is within a…

I'm fine with you running Anubis. My point was that Tavis flagged the outcome this blog post is discussing a year ago. It was hotly disputed on HN at the time. That's all I'm saying.

Re: Creepy Crawlies

#655

Post by [deleted] »

crote wrote: post #577
tptacek wrote: post #145

Earlier quoted context omitted.

I'm not moralizing. The sites running Anubis are owned by people who like Anubis and are fully within their rights to run Anubis. I don't care how enthusiastic they are about it. It's their call. I'm just saying, it's bad computer science.

The real world doesn't care about the soundness of your computer science. All that matters is whether it works. It's like saying we should just completely give up on all NP-hard problems because we'll never discover a way to find the optimal solution in all cases. Meanwhile in the real world, people are more than happy with, say, a route planner which virtually all of the time gives them a solution which is within a…

[deleted]

Re: Creepy Crawlies

#656

Post by Arubis »

alkonaut wrote: post #298
Arubis wrote: post #165

Earlier quoted context omitted.

How do you define humanity? How do you ensure it includes every human? How do you ensure it doesn’t include every non-human? I’m not even asking about computation or algorithms. I straight up don’t think you can make a definition that isn’t a tautology or an approximation. Both of which are useful, but neither of which can fit a _proof_.

Every place on earth has some legal definition of who is human. The system I’m thinking of isn’t a technical/captcha one, it’s a human curated list of humans. Just an electronic ID. Those already exist but the challenge is making them (acceptably) privacy-preserving. I want to take my existing national digital ID and use it online basically. BUT I don’t want the websites to know it’s me. Just that I’m human (or perha…

> Every place in earth has a legal definition of who is human.

I actually doubt that! And in places that do I doubt it’s perfection. Citizenship is mostly defined legally. Residency has a legal definition that may or may not track with reality. Most legal regimes I’m aware of (IANAL) define personhood; whether humanness is defined I’m less sure of. Things that are more nebulous and subjective…sometimes they’re defined, but necessarily subjectively so.

And that gets you into dangerous territory if you’re at the margins.

A fun example: how do you define “alive”? Does the person have to have been born? Are there conditions on that? Would they exclude some people we think of as alive? Does the person’s body need to be self-sufficient or can it rely on mechanical or other intervention? Is there a line? Is the person alive if their body is functioning but their brain is not? What about vice-versa? And even if the answers to all these questions are currently perfect, what happens if and when the definition shifts?

Being alive and human is messy business. And this is setting aside the discussion of making government issued records safely usable without leaking, which they already aren’t.

Re: Creepy Crawlies

#657

Post by timpera »

justAnotherHero wrote: post #404

While nowhere near compared to their scale, I run a consumer app where most of our users are using the mobile app, with the web app getting perhaps 10-15% of the mobile active users. However day after day it just gets blasted with requests for deep pages. I was quite alarmed when I saw a 100x increase in the daily active user numbers which relied on session length, only to realize they were all bots. Naively I too in…

I had the same problem, tried a bunch of stuff, banned millions of IPs, but nothing really worked. I ended up spending a bunch of time rewriting the website with Codex so that it is more efficient. Now I'm still getting the same junk traffic, but it barely affects the CPU anymore. I think this might be the way to go.

Re: Creepy Crawlies

#659

Post by voakbasda »

inigyou wrote: post #259
voakbasda wrote: post #71

Earlier quoted context omitted.

I would think that residential proxying would be illegal already, as it is a network intrusion. The trick is chasing down the offenders, proving their actions did harm, and getting them to pay. None of those steps are easy, even if there are laws to assist. Otherwise, spam would be a solved problem.

It isn't illegal, there is no law against "network intrusion" which is a term you just made up, and if it was a real term it probably wouldn't cover this. There are laws against things like "unauthorized access to a protected computer system".

Patayto, potahto. You care to explain how a “network intrusion” is not an “unauthorized access to a protected computer system”?

Re: Creepy Crawlies

#660

Post by done_lurking »

lxgr wrote: post #533
FridgeSeal wrote: post #471

Earlier quoted context omitted.

It doesn’t have to be fully sound, it’s there to provide _some_ degree of interim protection. Evidently, it makes _enough_ of a difference for the people running it. Criticising it is just letting the perfect be the enemy of the good. The alternative to Anubis isn’t “something better that isn’t being picked”, it’s “completely unprotected”, and the people criticising it for being imperfect could helpfully move the con…

Sure, but then you’re essentially just doing versions of the old “what is 2+2” text captchas with extra computational steps, so why not save yourself and your users those?

As gp said, it does provide some protection. If it didn't, nobody would use it and there would be nothing to discuss. Making snarky comparisons doesn't do anything.
Post reply on HN