Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Web Is Broken – Botnet Part 2

jan.wildeboer.net

Post reply on HN
Print view View original

211–220 of 301 posts

Re: The Web Is Broken – Botnet Part 2

#211

Post by areyourllySorry »

jeroenhd wrote: post #75
BlueTemplar wrote: post #42

Earlier quoted context omitted.

Yeah, also this means the death of archival efforts like the Internet Archive.

Welcome scrapers (IA, maybe Google and Bing) can publish their IP addresses and get whitelisted. Websites that want to prevent being on the Internet Archive can pretty much just ask for their website to be excluded (even retroactively). [Cloudflare]( https://developers.cloudflare.com/cache/troubleshooting/alwa... ) tags the internet archive as operating from 207.241.224.0/20 and 208.70.24.0/21 so disabling the bot-pr…

a large chunk of internet archive's snapshots are from archiveteam, where "warriors" bring their own ips (and they crawl respectfully!). save page now is important too, but you don't realise what is useful until you lose it.

Re: The Web Is Broken – Botnet Part 2

#212

Post by areyourllySorry »

0dayz wrote: post #168
realusername wrote: post #138

Earlier quoted context omitted.

That's basically asking to close the market in favor of the current actors. New actors have the right to emerge.

No they don't. There's no rule that you have to let anyone in who claims to be a web crawler.

which is why they will stop claiming to be one.

Re: The Web Is Broken – Botnet Part 2

#213

Post by areyourllySorry »

jeroenhd wrote: post #77

> So there is a (IMHO) shady market out there that gives app developers on iOS, Android, MacOS and Windows money for including a library into their apps that sells users network bandwidth AKA "why do Cloudflare and Google make me fill out these CAPTCHAs all day" I don't know why Play Protect/MS Defender/whatever Apple has for antivirus don't classify apps that embed such malware as such. It's ridiculous that this is…

it's not technically malware, you agreed to it when you accepted the terms of service :^)

Re: The Web Is Broken – Botnet Part 2

#214

Post by XorNot »

__MatrixMan__ wrote: post #195
XorNot wrote: post #171

Earlier quoted context omitted.

Except no one wants content addressed data - because if you knew what it was you wanted, then you would already have stored it. The web as we know it is an index - it's a way to discover that data is available and specifically we usually want the latest data that's available. AI scrapers aren't trying to find things they already know exist, they're trying to discover what they didn't know existed.

Yes, for the reasons you describe, you can't be both a useful web-like protocol and also 100% immutable/hash-linked. But there's a lot middle ground to explore here. Loading a modern web page involves making dozens of requests to a variety of different servers, evaluating some javascript, and then doing it again a few times, potentially moving several Mb of data. The part people want, the thing you don't already know…

But we already have HEAD requests and etags.

It is entirely possible to serve a fully cached response that says "you already have this". The problem is...people don't implement this well.

Re: The Web Is Broken – Botnet Part 2

#215

Post by areyourllySorry »

__MatrixMan__ wrote: post #93

The broken thing about the web is that in order for data to remain readable, a unique sysadmin somewhere has to keep a server running in the face of an increasingly hostile environment. If instead we had a content addressed model, we could drop the uniqueness constraint. Then these AI scrapers could be gossiping the data to one another (and incidentally serving it to the rest of us) without placing any burden on the…

there is no incentive for different companies to share data with each other, or with anyone really (facebook leeching books?)

Re: The Web Is Broken – Botnet Part 2

#218

Post by dspillett »

> So there is a (IMHO) shady market out there that gives app developers on iOS, Android, MacOS and Windows money for including a library into their apps that sells users network bandwidth.

This is yet another reason why we need to be wary of popular apps, add-ons, extensions, and so forth changing hands, by legitimate sale or more nefarious methods. Initially innocent utilities can be quickly coopted into being parts of this sort of scheme.

Re: The Web Is Broken – Botnet Part 2

#219

Post by CalRobert »

hubraumhugo wrote: post #178

We all agree that AI crawlers are a big issue as they don't respect any established best practices, but we rarely talk about the path forward. Scraping has been around for as long as the internet, and it was mostly fine. There are many very legitimate use cases for browser automation and data extraction (I work in this space). So what are potential solutions? We're somehow still stuck with CAPTCHAS, a 25 years old co…

I hate this but I suspect a login-only deanonymised web (made simple with chrome and WEI!) is the future. Firefox users can go to hell.

Re: The Web Is Broken – Botnet Part 2

#220

Post by mjmas »

udev4096 wrote: post #186
philippta wrote: post #79

Earlier quoted context omitted.

Apps should be required to ask for permission to access specific domains. Similar to the tracking protection, Apple introduced a while ago. Not sure how this could work for browsers, but the other 99% of apps I have on my phone should work fine with just a single permitted domain.

Android is so fucking anti-privacy that they still don't have an INTERNET access revoke toggle. The one they have currently is broken and can easily be bypassed with google play services (another highly privileged process running for no reason other than to sell your soul to google). GrapheneOS has this toggle luckily. Whenever you install an app, you can revoke the INTERNET access at the install screen and there is…

Asus added this to their phones which is nice.
Post reply on HN