Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Web Is Broken – Botnet Part 2

jan.wildeboer.net

Post reply on HN
Print view View original

261–270 of 301 posts

Re: The Web Is Broken – Botnet Part 2

#261

Post by [deleted] »

areyourllySorry wrote: post #215
__MatrixMan__ wrote: post #93

The broken thing about the web is that in order for data to remain readable, a unique sysadmin somewhere has to keep a server running in the face of an increasingly hostile environment. If instead we had a content addressed model, we could drop the uniqueness constraint. Then these AI scrapers could be gossiping the data to one another (and incidentally serving it to the rest of us) without placing any burden on the…

there is no incentive for different companies to share data with each other, or with anyone really (facebook leeching books?)

[deleted]

Re: The Web Is Broken – Botnet Part 2

#262

Post by __MatrixMan__ »

areyourllySorry wrote: post #215
__MatrixMan__ wrote: post #93

The broken thing about the web is that in order for data to remain readable, a unique sysadmin somewhere has to keep a server running in the face of an increasingly hostile environment. If instead we had a content addressed model, we could drop the uniqueness constraint. Then these AI scrapers could be gossiping the data to one another (and incidentally serving it to the rest of us) without placing any burden on the…

there is no incentive for different companies to share data with each other, or with anyone really (facebook leeching books?)

I figure we'd create that incentive by configuring our devices to only talk to devices controlled by people we trust. If they want the data at all, they have to gain our trust, and if they want that, they have to seed the data. Or you know, whatever else the agreement ends up being. Maybe we make them pay us.

Re: The Web Is Broken – Botnet Part 2

#263

Post by Imustaskforhelp »

rollcat wrote: post #232
aorth wrote: post #223

In the last week I've had to deal with two large-scale influxes of traffic on one particular web server in our organization. The first involved requests from 300,000 unique IPs in a span of a few hours. I analyzed them and found that ~250,000 were from Brazil. I'm used to using ASNs to block network ranges sending this kind of traffic, but in this case they were spread thinly over 6,000+ ASNs! I ended up blocking all…

Try Anubis: https://anubis.techaro.lol > It's a reverse proxy that presents a PoC challenge to every new visitor. It shifts the initial cost of accessing your server's resources back at the client. Assuming your uplink can handle 300k clients requesting a single 70kb web page, it should solve most of your problems. For science, can you estimate your peak QPS?

Also, I was just watching brodie robertson video about how United Nations has this random search page of unesco which actually has anubis.

Crazy how I remember the HN post where anubis's blog post was first made. Though, I always thought it was a bit funny with anime and it was made by frustration of (I think AWS? AI scrapers who won't follow general rules and it was constantly giving requests to his git server and it actually made his git server down I guess??) I didn't expect it to blow up to ... UN.

Re: The Web Is Broken – Botnet Part 2

#264

Post by xena »

Imustaskforhelp wrote: post #263
rollcat wrote: post #232

Earlier quoted context omitted.

Try Anubis: https://anubis.techaro.lol > It's a reverse proxy that presents a PoC challenge to every new visitor. It shifts the initial cost of accessing your server's resources back at the client. Assuming your uplink can handle 300k clients requesting a single 70kb web page, it should solve most of your problems. For science, can you estimate your peak QPS?

Also, I was just watching brodie robertson video about how United Nations has this random search page of unesco which actually has anubis. Crazy how I remember the HN post where anubis's blog post was first made. Though, I always thought it was a bit funny with anime and it was made by frustration of (I think AWS? AI scrapers who won't follow general rules and it was constantly giving requests to his git server and i…

Her*

It was frustration at AWS' Alexa team and their abuse of the commons. Amusingly if they had replied to my email before I wrote my shitpost of an implementation this all could have turned out vastly differently.

Re: The Web Is Broken – Botnet Part 2

#265

Post by zzo38computer »

kmeisthax wrote: post #127
klabb3 wrote: post #122

Earlier quoted context omitted.

On the one hand, yes this could work for many cases. On the other hand, good bye p2p. Not every app is a passive client-server request-response. One needs to be really careful with designing permission systems. Apple has already killed many markets before they had a chance to even exist, such as companion apps for watches and other peripherals.

P2P was practically dead on iPhone even back in 2010. The whole "don't burn the user's battery" thing precludes mobile phones doing anything with P2P other than leeching off of it. The only exceptions are things like AirDrop; i.e. locally peer-to-peer things that are only active when in use and don't try to form an overlay or mesh network that would require the phone to become a router. And, AFAIK, you already need s…

> other than HTTPS to specific domains on the public Internet

They should need special permission for that too.

Re: The Web Is Broken – Botnet Part 2

#266

Post by ClumsyPilot »

SoftTalker wrote: post #139
ClumsyPilot wrote: post #52

Earlier quoted context omitted.

> Have you audited this code? Wrong question. “Are you paid to audit this code?” And “if you fail to audit this code, who’se problem is it?”

If a car manufacturer sources a part from a third party, and that part has a serious safety problem, who will the customer blame? And who will be responsible for the recall and the repairs?

But we aren’t car business, am we are in joker business.

When was the last time producer of an app was held legally accountable for negligence, had to pay compensation and damages, etc?

Re: The Web Is Broken – Botnet Part 2

#267

Post by johnnyanmac »

CaptainFever wrote: post #224
hubraumhugo wrote: post #178

We all agree that AI crawlers are a big issue as they don't respect any established best practices, but we rarely talk about the path forward. Scraping has been around for as long as the internet, and it was mostly fine. There are many very legitimate use cases for browser automation and data extraction (I work in this space). So what are potential solutions? We're somehow still stuck with CAPTCHAS, a 25 years old co…

My pet peeve is that using the term "AI crawler" for this conflates things unnecessarily. There's some people who are angry at it due to anti-AI bias and not wishing to share information, while there are others who are more concerned about it due to the large amount of bandwidth and server overloading. Not to mention that it's unknown if these are actually from AI companies, or from people pretending to be AI compani…

It's become unbearable in the "AI era". So it's appropriate to blame AI for it, ib my eyes. Especially since so much defense is based aroind training LLMs.

It's just like how not all Ddoss's are actually hackers or bots. Sometimes a server just can't take the traffic of a large site flooding in. But the result is the same until something is investigated.

Re: The Web Is Broken – Botnet Part 2

#268

Post by hinkley »

nottorp wrote: post #255
hinkley wrote: post #252

Earlier quoted context omitted.

What’s that going to do about all of the top search results and a good percentage of social media traffic being generated by SEO bots? Nothing. You want to chat with a Dunbar number of people get yourself a private discord or slack channel.

The Dunbar number of people could vouch for small web sites they come across. Or even for FB accounts if they choose to.

I suspect a lot of people here are the ones in their circle who bring in a lot of the cool info that their friends missed out on. This still sounds like Slack.

Re: The Web Is Broken – Botnet Part 2

#269

Post by nottorp »

hinkley wrote: post #268
nottorp wrote: post #255

Earlier quoted context omitted.

The Dunbar number of people could vouch for small web sites they come across. Or even for FB accounts if they choose to.

I suspect a lot of people here are the ones in their circle who bring in a lot of the cool info that their friends missed out on. This still sounds like Slack.

We're talking about webs of trust aren't we? Not about chat rooms.

I'm hypothesising that any such large scale structure will be perverted by commercial interests, while having multiple Dunbar sized such structures will have a chance to be useful.

Re: The Web Is Broken – Botnet Part 2

#270

Post by queenkjuul »

CaptainFever wrote: post #224
hubraumhugo wrote: post #178

We all agree that AI crawlers are a big issue as they don't respect any established best practices, but we rarely talk about the path forward. Scraping has been around for as long as the internet, and it was mostly fine. There are many very legitimate use cases for browser automation and data extraction (I work in this space). So what are potential solutions? We're somehow still stuck with CAPTCHAS, a 25 years old co…

My pet peeve is that using the term "AI crawler" for this conflates things unnecessarily. There's some people who are angry at it due to anti-AI bias and not wishing to share information, while there are others who are more concerned about it due to the large amount of bandwidth and server overloading. Not to mention that it's unknown if these are actually from AI companies, or from people pretending to be AI compani…

It's not a coincidence that this wasn't a major problem until everybody and their dog started trying to build the next great LLM.
Post reply on HN