Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Web Is Broken – Botnet Part 2

jan.wildeboer.net

Post reply on HN
Print view View original

171–180 of 301 posts

Re: The Web Is Broken – Botnet Part 2

#171

Post by XorNot »

__MatrixMan__ wrote: post #93

The broken thing about the web is that in order for data to remain readable, a unique sysadmin somewhere has to keep a server running in the face of an increasingly hostile environment. If instead we had a content addressed model, we could drop the uniqueness constraint. Then these AI scrapers could be gossiping the data to one another (and incidentally serving it to the rest of us) without placing any burden on the…

Except no one wants content addressed data - because if you knew what it was you wanted, then you would already have stored it. The web as we know it is an index - it's a way to discover that data is available and specifically we usually want the latest data that's available.

AI scrapers aren't trying to find things they already know exist, they're trying to discover what they didn't know existed.

Re: The Web Is Broken – Botnet Part 2

#172

Post by reconnecting »

sroussey wrote: post #74
reconnecting wrote: post #56

Earlier quoted context omitted.

Indeed, if it's a real user from a residential IP address, in most cases it will be the same network. However, if it's a proxy from residential IPs, there could be 10 requests from one network, the 11th request from a second network, and the 12th request back from the same network. This is a red flag. Thank you for your question. tirreno is a standalone app that needs to receive API events from your main web applicat…

My phone will be on the home network until I walk out of the house and then it will change networks. This should not be a red flag.

Effective fraud prevention relies on both the full user context and the behavioral patterns of known online fraudsters. The key idea is that an IP address cannot be used as a red flag on its own without considering the broader context of the account. However, if we know that the fraudsters we're dealing with are using mobile networks proxies and are randomly switching between two mobile operators, that is certainly a strong risk signal.

Re: The Web Is Broken – Botnet Part 2

#173

Post by dewey »

gruez wrote: post #111
abaymado wrote: post #86

Earlier quoted context omitted.

> iOS have app privacy report where one can check what connections are made by app, how often, last one, etc. How often is the average calculator app user checking there Privacy Report? My guess, not many!

All it takes is one person to find out and raise the alarm. The average user doesn't read the source code behind openssl or whatever either, that doesn't mean there's no gains in open sourcing it.

The average user is also not reading these raised “alarms”. And if an app has a bad name, another one will show up with a different name on the same day.

Re: The Web Is Broken – Botnet Part 2

#174

Post by akoboldfrying »

hinkley wrote: post #165

When the enshitification initially hit the fan, I had little flashbacks of Phil Zimmerman talking about Web of Trust and amusing myself thinking maybe we need humans proving they're humans to other humans so we know we aren't arguing with LLMs on the internet or letting them scan our websites. But it just doesn't scale to internet size so I'm fucked if I know how we should fix it. We all have that cousin or dude in o…

I think it should be possible to build something that generalises the idea of Web of Trust so that it's more flexible, and less prone to catastrophic breakdown past some scaling limit.

Binary "X trusts Y" statements, plus transitive closure, can lead to long trust paths that we probably shouldn't actually trust the endpoints of. Could we not instead assign probabilities like "X trusts Y 95%", multiply probabilities along paths starting from our own identity, and take the max at each vertex? We could then decide whether to finally trust some Z if its percentage is more than some threshold T%. (Other ways of combining in-edges may be more suitable than max(); it's just a simple and conservative choice.)

Perhaps a variant of backprop could be used to automatically update either (a) all or (b) just our own weights, given new information ("V has been discovered to be fraudulent").

Re: The Web Is Broken – Botnet Part 2

#175

Post by dx4100 »

jeroenhd wrote: post #77

> So there is a (IMHO) shady market out there that gives app developers on iOS, Android, MacOS and Windows money for including a library into their apps that sells users network bandwidth AKA "why do Cloudflare and Google make me fill out these CAPTCHAs all day" I don't know why Play Protect/MS Defender/whatever Apple has for antivirus don't classify apps that embed such malware as such. It's ridiculous that this is…

Cloudflare and Google use CAPTCHAs to sell web scrapers? I don't get your point. I was under the impression the data is used to train models.

Re: The Web Is Broken – Botnet Part 2

#176

Post by Centigonal »

zahlman wrote: post #13

> I am now of the opinion that every form of web-scraping should be considered abusive behaviour and web servers should block all of them. If you think your web-scraping is acceptable behaviour, you can thank these shady companies and the “AI” hype for moving you to the bad corner. I imagine that e.g. Youtube would be happy to agree with this. Not that it would turn them against AI generally.

yeah, but you can't, that's the problem. Plenty of service operators would like to block every scraper that doesn't obey their robots.txt, but there's no good way to do that without blocking human traffic too (Anubis et al are okay, but they are half-measures).

On a separate note, I believe open web scraping has been a massive benefit to the internet on net, and almost entirely positive pre-2021. Web scraping & crawling enables search engines, services like Internet Archive, walled-garden-busting (like Invidious, yt-dlp, and Nitter), mashups (Spotube, IFTT, and Plaid would have been impossible to bootstrap without web scraping), and all kinds of interesting data science projects (e.g. scraping COVID-19 stats from local health departments to patch together a picture of viral spread for epidemiologists).

Re: The Web Is Broken – Botnet Part 2

#177

Post by akoboldfrying »

Liftyee wrote: post #4

I don't know if I should be surprised about what's described in this article, given the current state of the world. Certainly I didn't know about it before, and I agree with the article's conclusion. Personally, I think the "network sharing" software bundled with apps should fall into the category of potentially unwanted applications along with adware and spyware. All of the above "tag along" with something the user…

I agree, but the harm done to the users is only one part of the total harm. I think it's quite plausible that many users wouldn't mind some small amount of their bandwidth being used, if it meant being able to use a handy browser extension that they would otherwise have to pay actual dollars for -- but the harm done to those running the servers remains.

Re: The Web Is Broken – Botnet Part 2

#178

Post by hubraumhugo »

We all agree that AI crawlers are a big issue as they don't respect any established best practices, but we rarely talk about the path forward. Scraping has been around for as long as the internet, and it was mostly fine. There are many very legitimate use cases for browser automation and data extraction (I work in this space).

So what are potential solutions? We're somehow still stuck with CAPTCHAS, a 25 years old concept that wastes millions of human hours and billions in infra costs [0].

How can enable beneficial automation while protecting against abusive AI crawlers?

[0] https://arxiv.org/abs/2311.10911

Re: The Web Is Broken – Botnet Part 2

#179

Post by akoboldfrying »

XorNot wrote: post #171
__MatrixMan__ wrote: post #93

The broken thing about the web is that in order for data to remain readable, a unique sysadmin somewhere has to keep a server running in the face of an increasingly hostile environment. If instead we had a content addressed model, we could drop the uniqueness constraint. Then these AI scrapers could be gossiping the data to one another (and incidentally serving it to the rest of us) without placing any burden on the…

Except no one wants content addressed data - because if you knew what it was you wanted, then you would already have stored it. The web as we know it is an index - it's a way to discover that data is available and specifically we usually want the latest data that's available. AI scrapers aren't trying to find things they already know exist, they're trying to discover what they didn't know existed.

> because if you knew what it was you wanted, then you would already have stored it.

"Content-addressable" has a broader meaning than what you seem to be thinking of -- roughly speaking, it applies if any function of the data is used as the "address". E.g., git commits are content-addressable by their SHA1 hashes.

Re: The Web Is Broken – Botnet Part 2

#180

Post by Galanwe »

aucisson_masque wrote: post #60

It's interesting but so far there is no definitive proof it's happening. People are jumping to conclusions a bit fast over here, yes technically it's possible but this kind of behavior would be relatively easy to spot because the app would have to make direct connections to the website it wants to scrap. Your calculator app for instance connecting to CNN.com ... iOS have app privacy report where one can check what co…

There is already a lot of proof. Just ask for a sales pitch from companies selling these data and they will gladly explain everything to you.

Go to a data conference like Neudata and you will see. You can have scraped data from user devices, real-time locations, credit card, Google analytics, etc.

Post reply on HN