Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

91–100 of 683 posts

Re: Creepy Crawlies

#91

Post by tmp10423288442 »

ipdashc wrote: post #87
rcxdude wrote: post #40

Earlier quoted context omitted.

These crawlers (in contrast to e.g. googlebot and similar better behaved crawlers) are not very smart: they seem to make very little effort to avoiding crawling useless deep trees of generated pages. About the only thing they seem to put a lot of effort into is avoiding blocking. (I'll note that while these are generally attributed to AI data gathering because of the timing of when they took off, it's not actually ob…

> it's not actually obvious who's running these bots This is fascinating to me. It's a large enough phenomenon that it's affecting the entire Internet and yet nobody seems to know yet who's actually doing it. Which isn't surprising, of course, it's hard to trace back to a source through all these proxies and it's probably a bunch of distinct groups anyways, but still! Personally I have to wonder how much of it is "sc…

ChatGPT uses a distinct bot type for ChatGPT-initiated web search versus model training:

GPTBot → model improvement / training crawl

OAI-SearchBot → search index / ChatGPT Search

ChatGPT-User → fetch initiated by a user's ChatGPT interaction

OAI-AdsBot → advertising landing-page crawl

Re: Creepy Crawlies

#92

Post by smallerize »

semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

But the scraper is making way more requests and is paying for all that compute.

Re: Creepy Crawlies

#94

Post by feelamee »

klez wrote: post #47
feelamee wrote: post #41

Hm, interesting - how will it look the actual solution for such problems in the future. I suppose the issue will continue to grow. First idea - there should be some cost for sending traffic somewhere. And the server owner also should receive pay - not only the internet provider. So, in with this idea, the server owner can potentially increase the amount of computing power to satisfy all requests.

> there should be some cost for sending traffic somewher So now, because of bad actors, I need to pay for the privilege of watching a website or using a service that was meant to be free? No, thanks. I don't have a solution, but "break how the web currently works" is not one I would accept all willy nilly. EDIT: yes, I realize we already broke the web (with Anubis, cloudflare, recaptcha etc) but I think we should res…

> So now, because of bad actors, I need to pay for the privilege of watching a website or using a service that was meant to be free? No, thanks.

First of all - I suppose it should be very cheap. So, real humans will not pay much.

Second - why do u think that websites are meant to be free? They provide some service, so its a rather strange that the internet is so free (in both senses). I think, this freeiness is allowed to greatly speed up popularization. But for me is obvious that it can demand payment for service.

And third - service owner really meant it to be free, I don't see any problems with this in my idea. It can still provide free service.

Re: Creepy Crawlies

#97

Post by Demiurge »

klez wrote: post #35
rankdiff wrote: post #32

Earlier quoted context omitted.

What is "formally popular"?

From context I'd say they meant "formerly" and that's just a typo. I didn't even notice the error before you pointed it out.

Yes, my bad! Typing on the terrible iPhone keyboard :)

Re: Creepy Crawlies

#98

Post by feelamee »

rwmj wrote: post #65
feelamee wrote: post #41

Hm, interesting - how will it look the actual solution for such problems in the future. I suppose the issue will continue to grow. First idea - there should be some cost for sending traffic somewhere. And the server owner also should receive pay - not only the internet provider. So, in with this idea, the server owner can potentially increase the amount of computing power to satisfy all requests.

It's a social problem, technical solutions aren't going to work. To solve social problems you need the law to get involved. It should simply be illegal to install malware on someone's TV and use it to hijack their internet connection, and the punishment should be the same as for any theft of service, including imprisonment for people who do it.

The law will not work without a technical solution. So, you need both. But, at least, technical.

Re: Creepy Crawlies

#99

Post by marginalia_nu »

wiredfool wrote: post #90
delichon wrote: post #42

> Why is git.kernel.org “interesting” to crawlers Interesting to crawlers is not a narrow scope. We have the same problem on a B2B car wash site.

Seeing the exact same thing on (somewhat high profile) open data sites I run. The crawlers get stuck in a loop requesting the dataset listing page with every. single. combination. of. facets. At essentially as fast as it can be pumped out or blocked. Had one bot super interested in a single organization to the tune of 1000 r/min for 24+ hours. Realistic rotating user agents, realistic sec-* headers, no ip address see…

Yeah my search engine saw traffic of up to 160 queries per second the other day from some bot that was ostensibly searching for information on Jack Parsons. Just variations on the same query in different permutations of filters and site:-terms.

Re: Creepy Crawlies

#100

Post by wongarsu »

semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

Isn't the issue here the three order of magnitude difference between the (presumably optimized) JS implementation and the optimized C kernel on your browser? If the two stay within an order of magnitude of each other Anubis is a perfectly viable and scalable solution. Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them

For reference, the challenge on lists.ffmpeg.org takes 8 seconds on Firefox on my three year old laptop CPU that has worse benchmark scores than the iPhone 17 (tbf, the laptop also cost less than an iPhone 17). 8 seconds doesn't run against thermal limitations, so I really don't see why Safari on a modern iPhone should be so slow at this

Post reply on HN