Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

261–270 of 691 posts

Re: Creepy Crawlies

#261

Post by inigyou »

marginalia_nu wrote: post #99
wiredfool wrote: post #90

Earlier quoted context omitted.

Seeing the exact same thing on (somewhat high profile) open data sites I run. The crawlers get stuck in a loop requesting the dataset listing page with every. single. combination. of. facets. At essentially as fast as it can be pumped out or blocked. Had one bot super interested in a single organization to the tune of 1000 r/min for 24+ hours. Realistic rotating user agents, realistic sec-* headers, no ip address see…

Yeah my search engine saw traffic of up to 160 queries per second the other day from some bot that was ostensibly searching for information on Jack Parsons. Just variations on the same query in different permutations of filters and site:-terms.

That would be a different bot, one written specifically for your site. Mainly we're discussing the dumb ones that just crawl all possible http links

Re: Creepy Crawlies

#262

Post by rplnt »

Aurornis wrote: post #187
wongarsu wrote: post #100

Earlier quoted context omitted.

Isn't the issue here the three order of magnitude difference between the (presumably optimized) JS implementation and the optimized C kernel on your browser? If the two stay within an order of magnitude of each other Anubis is a perfectly viable and scalable solution. Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them Fo…

> Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them Depending on the configuration, Anubis will supply a token after the challenge that bypasses the challenge for a time. So any scraper that retains basic cookies will be able to bypass the challenge for a number of page views. A user who needs to load a single page and…

Now you have a session of sorts and can limit the requests for that client, right? They can be fast, just limited in volume - regular user isn't punished.

Re: Creepy Crawlies

#263

Post by inigyou »

__MatrixMan__ wrote: post #199

This is a fundamental flaw in the web. Since we treat a server's name as authoritative, anybody maintaining a replica needs to repeatedly hit that server to know if their remote version is up to date. If we trusted digital signatures on content instead of server names, we could have a model where a single bit of server load propagates to millions of interested parties. As it is the server must do something distinct f…

Or poll a merkle tree root hash, like, um, git

Re: Creepy Crawlies

#264

Post by TonyTrapp »

mzajc wrote: post #201

> Why is git.kernel.org “interesting” to crawlers I think the post underestimates just how little thought and effort is put into these bots. I also run a cgit instance with far less interesting projects, and am not spared from the deluge of HTTP requests. The explanation I could come up with is that they try to crawl all links regardless of how much sense it makes or how much load it causes. cgit being cgit, this mea…

Exactly my observation as well. They devour absolutely everything, no exceptions. No matter how stupid it might be to digest a source code repository via HTTP. They probably don't even recognize what's inside those pages and that there's an easier way to obtain the same result.

Re: Creepy Crawlies

#266

Post by inigyou »

calvinmorrison wrote: post #249

I love to see the 'leet kernel hackers and maintainers' struggling with basic volume. Each page load should cost you near nil. Us lowly PHP developers have been caching shit for close to twenty years. Learn how to cache your application and your cpu usage should be almost zero. In fact basically any read should cost nothing in comparison to writes.

It's running a diff between two arbitrary blobs of text. Do you actually have a solution or are you just saying to remove the feature from the site entirely?

> Us PHP developers

I can tell.

Re: Creepy Crawlies

#267

Post by TeMPOraL »

inigyou wrote: post #238
tptacek wrote: post #145

Earlier quoted context omitted.

I'm not moralizing. The sites running Anubis are owned by people who like Anubis and are fully within their rights to run Anubis. I don't care how enthusiastic they are about it. It's their call. I'm just saying, it's bad computer science.

Which part of computer science is about getting DDoSed by billionaires who are above the law?

Where on Earth do people get the belief that:

- It's the SOTA companies doing it?

- Scrapers are doing it for training data?

Those are two assumptions I see in posts and threads around Anubis, that are taken at faith, and never once substantiated.

Re: Creepy Crawlies

#268

Post by inigyou »

mbirth wrote: post #194
mattmcal wrote: post #181

Earlier quoted context omitted.

There is a section in the article answering your question if you read it. > Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again. There was no point in banning them, because by the time you figured out that they were bots, they were already done with you.

I’d love a service like spamcop.net where I could submit my access_log and they lookup the abuse addresses and file abuse reports in my name. Maybe if people’s Internet access gets suspended they’ll think about installing random apps that work as a proxy in the background.

abuseipdb.com

Some ISPs ban customers based on a single report there - have fun!

Re: Creepy Crawlies

#269

Post by inigyou »

klez wrote: post #47
feelamee wrote: post #41

Hm, interesting - how will it look the actual solution for such problems in the future. I suppose the issue will continue to grow. First idea - there should be some cost for sending traffic somewhere. And the server owner also should receive pay - not only the internet provider. So, in with this idea, the server owner can potentially increase the amount of computing power to satisfy all requests.

> there should be some cost for sending traffic somewher So now, because of bad actors, I need to pay for the privilege of watching a website or using a service that was meant to be free? No, thanks. I don't have a solution, but "break how the web currently works" is not one I would accept all willy nilly. EDIT: yes, I realize we already broke the web (with Anubis, cloudflare, recaptcha etc) but I think we should res…

The internet is already like that, but for some reason the payment only extends as far as the recipient's ISP, not the actual recipient. Most senders pay a flat rate, but their ISP doesn't.

Re: Creepy Crawlies

#270

Post by applfanboysbgon »

wbl wrote: post #232
inigyou wrote: post #196

Earlier quoted context omitted.

And without control of their source IP addresses. They use a "rotating proxy" meta-proxy which sends each request through a different proxy, which invalidates the Anubis response and triggers a new challenge.

Real attackers fuck. If they have IP address control issues they will fix them. The bots being stopped are the silly ones.

Okay. That's good enough. The problem people using Anubis have is not "I'm being targeted for a DDoS" and is actually "my site is being incidentally DDoS'd because scrapers are DDoSing the entire internet". A solution to the latter is what people need. Every single one of these threads comes up with hypotheticals about how useless Anubis is and how it can't possibly work and yet it does work for the real-world problem that real-world webmasters are dealing with.
Post reply on HN