Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

201–210 of 687 posts

Re: Creepy Crawlies

#201

Post by mzajc »

> Why is git.kernel.org “interesting” to crawlers

I think the post underestimates just how little thought and effort is put into these bots. I also run a cgit instance with far less interesting projects, and am not spared from the deluge of HTTP requests.

The explanation I could come up with is that they try to crawl all links regardless of how much sense it makes or how much load it causes. cgit being cgit, this means billions of links for all combinations of parameters and hashes. That, or it's a deliberate DDoS attack.

Re: Creepy Crawlies

#202

Post by DarmokTanagra »

smallerize wrote: post #92
semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

But the scraper is making way more requests and is paying for all that compute.

not if its running on someone else's device

Re: Creepy Crawlies

#203

Post by paytonjjones »

pbronez wrote: post #193
paytonjjones wrote: post #157

Earlier quoted context omitted.

In this case, because there's a vastly more efficient economic path for the adversaries (cloning). They're not trying to engage in an arms race, they're trying to channel a racing river into its natural course.

It’s more economical at a compute level, but not at the developer level. The moment you start customizing your crawler to use protocol X for site Y your scale story collapses.

It's a good point, but in practice it depends on how easy those customizations are to implement / maintain, and how much money and effort you save. At some point the compute cost can disrupt even the nicest scale story.

I think the path forward is that websites offer one path for humans, and another for scrapers. But the huge catch is the path for scrapers must be _genuinely_ and _reliably_ the more economical and scalable path (either through something like PoW arms races, or through fear of litigation). Otherwise they will continue to ignore instructions and intrude on the human path.

Re: Creepy Crawlies

#204

Post by dennis-tra »

__MatrixMan__ wrote: post #199

This is a fundamental flaw in the web. Since we treat a server's name as authoritative, anybody maintaining a replica needs to repeatedly hit that server to know if their remote version is up to date. If we trusted digital signatures on content instead of server names, we could have a model where a single bit of server load propagates to millions of interested parties. As it is the server must do something distinct f…

Content-addressing decouples the hoster from the data itself. Anyone can serve content-addressed data and you can locally verify that you’ve got served the correct bytes. IPFS implements building blocks for such an alternative web.

What irony that this article is about crawling content-addressed data.

Re: Creepy Crawlies

#205

Post by arlattimore »

In the case of kernel.org, why not make the unauthenticated version return only the latest kernel repo with no history (tiny number of URLs relatively speaking). If you want full kernel.org features, login.

Re: Creepy Crawlies

#206

Post by __MatrixMan__ »

dennis-tra wrote: post #204
__MatrixMan__ wrote: post #199

This is a fundamental flaw in the web. Since we treat a server's name as authoritative, anybody maintaining a replica needs to repeatedly hit that server to know if their remote version is up to date. If we trusted digital signatures on content instead of server names, we could have a model where a single bit of server load propagates to millions of interested parties. As it is the server must do something distinct f…

Content-addressing decouples the hoster from the data itself. Anyone can serve content-addressed data and you can locally verify that you’ve got served the correct bytes. IPFS implements building blocks for such an alternative web. What irony that this article is about crawling content-addressed data.

Agreed on both points. But it's looking increasingly likely that particular dream is not coming true. Kubo, the reference implementation IPFS node, is maintainerless as of last week.

I'm not sure where IPFS went wrong, but I think our web will continue to degrade until we figure it out. (It'll continue to degrade after that also, but then we can let it burn since we'll have a replacement to switch to).

Re: Creepy Crawlies

#207

Post by minimaltom »

jwilk wrote: post #178
radlad wrote: post #132

Earlier quoted context omitted.

I'm reminded of LUKS Argon2 implementation, which is much slower than userspace. Therefore a sufficiently difficult challenge that may take less than a second in userspace Linux can take 20 or more seconds at boot.

Huh? Are you saying there's an Argon2 implementation in the kernel?

Its not in the kernel but in the userspace tool that goes from password to key (the key is handed to the kernel).

You can see the implementation here: https://gitlab.com/cryptsetup/cryptsetup/-/blob/main/lib/cry...

Re: Creepy Crawlies

#208

Post by __MatrixMan__ »

__MatrixMan__ wrote: post #206
dennis-tra wrote: post #204

Earlier quoted context omitted.

Content-addressing decouples the hoster from the data itself. Anyone can serve content-addressed data and you can locally verify that you’ve got served the correct bytes. IPFS implements building blocks for such an alternative web. What irony that this article is about crawling content-addressed data.

Agreed on both points. But it's looking increasingly likely that particular dream is not coming true. Kubo, the reference implementation IPFS node, is maintainerless as of last week. I'm not sure where IPFS went wrong, but I think our web will continue to degrade until we figure it out. (It'll continue to degrade after that also, but then we can let it burn since we'll have a replacement to switch to).

I think focusing on filecoin was probably the mistake. You've got to build something that people trust first and then consider adding a money-shaped app. If you start with something money-shaped you're indistinguishable from the legions of scams, and that's a hard position to start from if you're wanting to build something trustworthy.

Re: Creepy Crawlies

#209

Post by Arubis »

Kamq wrote: post #191
Arubis wrote: post #165

Earlier quoted context omitted.

How do you define humanity? How do you ensure it includes every human? How do you ensure it doesn’t include every non-human? I’m not even asking about computation or algorithms. I straight up don’t think you can make a definition that isn’t a tautology or an approximation. Both of which are useful, but neither of which can fit a _proof_.

You're not wrong, but something can work well enough to still be useful despite falling short of the idea of a proof or any formal definition. Let's say that 95% of individual humans can pass it and only 2% of bots. For someone maintaining a website, who has to decide between using this system and shutting down their site because of the increased costs, that may very well be good enough

That’s a pragmatic and understandable argument. And for an individual hobbyist site owner, that’s fine. Are we okay with excluding 1 person in 20 from the services of a midsized organization? What if they’re integral to the workplace? Or a major transport provider without differentiated competitors? What if the organization is a state government?

Re: Creepy Crawlies

#210

Post by Symbiote »

andruby wrote: post #117

A creepy crawly is a South African invention to clean your swimming pool. The company that introduced them in the 70ies is called Kreepy Krauly. Also popular in Australia. https://kreepykrauly.co.za/about-us/

It's a childish word for an insect.
Post reply on HN