Let me get this straight: we want computers knowing everything, to solve current and future problems, but we don't want to give them access to our knowledge?
I don't want your computer to know everything about me, in fact.
The Web Is Broken – Botnet Part 2
191–200 of 301 posts
Re: The Web Is Broken – Botnet Part 2
#192Earlier quoted context omitted.
All it takes is one person to find out and raise the alarm. The average user doesn't read the source code behind openssl or whatever either, that doesn't mean there's no gains in open sourcing it.
The average user is also not reading these raised “alarms”. And if an app has a bad name, another one will show up with a different name on the same day.
You're on a tech forum, you must have seen one of the many post about app, either on Android or iPhone, that acts like spyware.
They happens from time to time, last one was not more than two week ago where it's been shown that many app were able to read the list of all other app installed on a Android and that Google refused to fix that.
Do you really believe that an app used to make your device part of a bot network wouldn't be posted over here ?
Re: The Web Is Broken – Botnet Part 2
#193Earlier quoted context omitted.
Cloudflare and Google use CAPTCHAs to sell web scrapers? I don't get your point. I was under the impression the data is used to train models.
The implication is that the users that are being constantly presented with CAPTCHAs are experiencing that because they are unwittingly proxying scrapers through their devices via malicious apps they've installed.
.. or that other people on their network/Shared public IP have installed
Re: The Web Is Broken – Botnet Part 2
#194Let me get this straight: we want computers knowing everything, to solve current and future problems, but we don't want to give them access to our knowledge?
Most people don’t want computers to know everything - ask the average person if they want more or less of their lives recorded and stored.
Re: The Web Is Broken – Botnet Part 2
#195The broken thing about the web is that in order for data to remain readable, a unique sysadmin somewhere has to keep a server running in the face of an increasingly hostile environment. If instead we had a content addressed model, we could drop the uniqueness constraint. Then these AI scrapers could be gossiping the data to one another (and incidentally serving it to the rest of us) without placing any burden on the…
Except no one wants content addressed data - because if you knew what it was you wanted, then you would already have stored it. The web as we know it is an index - it's a way to discover that data is available and specifically we usually want the latest data that's available. AI scrapers aren't trying to find things they already know exist, they're trying to discover what they didn't know existed.
Yes, for the reasons you describe, you can't be both a useful web-like protocol and also 100% immutable/hash-linked.
But there's a lot middle ground to explore here. Loading a modern web page involves making dozens of requests to a variety of different servers, evaluating some javascript, and then doing it again a few times, potentially moving several Mb of data. The part people want, the thing you don't already know exist, it's hidden behind that rather heavy door. It doesn't have to be that way.
If you already know about one thing (by its cryptographic hash, say) and you want to find out which other hashes it's now associated with--associations that might not have existed yesterday--that's much easier than we've made it. It can be done:
- by moving kB not Mb, we're just talking about a tuple of hashes here, maybe a public key and a signature
- without placing additional burden on whoever authored the first thing, they don't even have to be the ones who published the pair of hashes that your scraper is interested in
Once you have the second hash, you can then reenter immutable-space to get whatever it references. I'm not sure if there's already a protocol for such things, but if not then we can surely make one that's more efficient and durable than what we're doing now.
Re: The Web Is Broken – Botnet Part 2
#196Earlier quoted context omitted.
The implication is that the users that are being constantly presented with CAPTCHAs are experiencing that because they are unwittingly proxying scrapers through their devices via malicious apps they've installed.
.. or that other people on their network/Shared public IP have installed
or just that they don't run windows/mac OS with chome like everyone else and it's "suspicious".
I get cloudflare capchas all the time with firefox on linux... (and I'm pretty sure there's no such app in my home network!)
Re: The Web Is Broken – Botnet Part 2
#197Earlier quoted context omitted.
The average user is also not reading these raised “alarms”. And if an app has a bad name, another one will show up with a different name on the same day.
You're on a tech forum, you must have seen one of the many post about app, either on Android or iPhone, that acts like spyware. They happens from time to time, last one was not more than two week ago where it's been shown that many app were able to read the list of all other app installed on a Android and that Google refused to fix that. Do you really believe that an app used to make your device part of a bot network…
"You're on a tech forum", that's exactly the point. The "average user" is not on a tech forum though, the average user opens the app store of their platform, types "calculator" and installs the first one that's free.
Re: The Web Is Broken – Botnet Part 2
#198We all agree that AI crawlers are a big issue as they don't respect any established best practices, but we rarely talk about the path forward. Scraping has been around for as long as the internet, and it was mostly fine. There are many very legitimate use cases for browser automation and data extraction (I work in this space). So what are potential solutions? We're somehow still stuck with CAPTCHAS, a 25 years old co…
Blame the "AI" companies for that. I am glad the small web is pushing hard against these scrapers, with the rise of Anubis as a starting point
> Blame the "AI" companies for that. I am glad the small web is pushing hard towards these scrapers, with the rise of Anubis as a starting point
Did you mean "against"?
Re: The Web Is Broken – Botnet Part 2
#199> I am now of the opinion that every form of web-scraping should be considered abusive behaviour and web servers should block all of them. If you think your web-scraping is acceptable behaviour, you can thank these shady companies and the “AI” hype for moving you to the bad corner. I imagine that e.g. Youtube would be happy to agree with this. Not that it would turn them against AI generally.
yeah, but you can't, that's the problem. Plenty of service operators would like to block every scraper that doesn't obey their robots.txt, but there's no good way to do that without blocking human traffic too (Anubis et al are okay, but they are half-measures). On a separate note, I believe open web scraping has been a massive benefit to the internet on net, and almost entirely positive pre-2021. Web scraping & crawl…
> Plenty of service operators would like to block every scraper that doesn't obey their robots.txt, but there's no good way to do that without blocking human traffic too (Anubis et al are okay, but they are half-measures)
Why is Anubis-type mitigations a half-measure?
Re: The Web Is Broken – Botnet Part 2
#200We all agree that AI crawlers are a big issue as they don't respect any established best practices, but we rarely talk about the path forward. Scraping has been around for as long as the internet, and it was mostly fine. There are many very legitimate use cases for browser automation and data extraction (I work in this space). So what are potential solutions? We're somehow still stuck with CAPTCHAS, a 25 years old co…
But people don’t interact with your website anymore; they as an AI. So the AI crawler is a real user.
I say we ask Google Analytics to count an AI crawler as a real view. Let’s see who’s most popular.