Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Web Is Broken – Botnet Part 2

jan.wildeboer.net

Post reply on HN
Print view View original

271–280 of 301 posts

Re: The Web Is Broken – Botnet Part 2

#271

Post by caelinsutch »

hubraumhugo wrote: post #178

We all agree that AI crawlers are a big issue as they don't respect any established best practices, but we rarely talk about the path forward. Scraping has been around for as long as the internet, and it was mostly fine. There are many very legitimate use cases for browser automation and data extraction (I work in this space). So what are potential solutions? We're somehow still stuck with CAPTCHAS, a 25 years old co…

CAPTCHAS are also quickly becoming irrelevant / not enough. Fingerprint based approaches seem to be the only realistic way forward in the cat / mouse game

Re: The Web Is Broken – Botnet Part 2

#272

Post by throwaway74663 »

reincoder wrote: post #260

I work for IPinfo (a commercial service). We offer a residential proxy detection service, but it costs money. If you are being bombarded by suspicious IP addresses, please consider using our free service and blocking IP addresses by ASN or Country. I think ASN is a common parameter for malicious IP addresses. If you do not have time to explore our services/tools (it is mostly just our CLI: https://github.com/ipinfo/c…

Blocking countries is such a poorly disguised form of racism. Funny how it's always the brown / yellow people countries that get blocked, and never the US, despite it being one of the leading nations in malicious traffic.

Re: The Web Is Broken – Botnet Part 2

#273

Post by akoboldfrying »

__MatrixMan__ wrote: post #258
akoboldfrying wrote: post #187

Earlier quoted context omitted.

Assuming the right incentives can be found to prevent widespread leeching, a distributed content-addressed model indeed solves this problem, but introduces the problem of how to control your own content over time. How do you get rid of a piece of content? How do you modify the content at a given URL? I know, as far as possible it's a good idea to have content-immutable URLs. But at some point, I need to make www.myex…

As for how to get rid of a piece of content... I think that one's a lost cause. If the goal is to prevent things that make content unavailable (e.g. AI scrapers) then you end up with a design that prevents things that makes content unavailable (e.g. legitimate deletions). The whole point is that you're not the only one participating in propagating the content, and that comes with trade-offs. But as for updating, you…

> But then again, could you ever do that?

True in the absolute sense, but the effect size is much worse under the kind of content-addressable model you're proposing. Currently, if I download something from you and you later delete that thing, I can still keep my downloaded copy; under your model, if anyone ever downloads that thing from you and you later delete that thing, with high probability I can still acquire it at any later point.

As you say, this is by design, and there are cases where this design makes sense. I think it mostly doesn't for what we currently use the web for.

Re: The Web Is Broken – Botnet Part 2

#274

Post by rjbwork »

morsecodist wrote: post #145
squeaky-clean wrote: post #128

Earlier quoted context omitted.

They're not really exclusive ideas. The Constructor Injection section in Fowler's article is exactly the same as the Strategy pattern. But no one talks about the Strategy pattern anymore, it's all wrapped into the idea of DI and that's what caught on.

It was interesting reading this exchange. I have a similar understanding of DI to you. I have never even heard of a DI framework and I have trouble picturing what it would look like. It was interesting to watch you two converge on where the disconnect was.

Usually when people refer to "DI Frameworks" they're referring to Inversion of Control (IoC) containers.

Re: The Web Is Broken – Botnet Part 2

#275

Post by __MatrixMan__ »

akoboldfrying wrote: post #273
__MatrixMan__ wrote: post #258

Earlier quoted context omitted.

As for how to get rid of a piece of content... I think that one's a lost cause. If the goal is to prevent things that make content unavailable (e.g. AI scrapers) then you end up with a design that prevents things that makes content unavailable (e.g. legitimate deletions). The whole point is that you're not the only one participating in propagating the content, and that comes with trade-offs. But as for updating, you…

> But then again, could you ever do that? True in the absolute sense, but the effect size is much worse under the kind of content-addressable model you're proposing. Currently, if I download something from you and you later delete that thing, I can still keep my downloaded copy; under your model, if anyone ever downloads that thing from you and you later delete that thing, with high probability I can still acquire it…

You could only later get the thing if you grabbed its hash while it was still available. And you could only reliably resolve that hash later if somebody (maybe you) went out of their way to pin the underlying data. Otherwise nodes would forget rather quickly, because why bother keep around unreferenced bits?

It's the same functionality you get with permalinks and sites like archive.org--forgotten unless explicitly remembered by anybody, dynamic unless explicitly a permalink. It's just built into the protocol rather than a feature to be inconsistently implemented over and over by many separate parties.

Re: The Web Is Broken – Botnet Part 2

#276

Post by ninkendo »

luckylion wrote: post #234
aorth wrote: post #223

In the last week I've had to deal with two large-scale influxes of traffic on one particular web server in our organization. The first involved requests from 300,000 unique IPs in a span of a few hours. I analyzed them and found that ~250,000 were from Brazil. I'm used to using ASNs to block network ranges sending this kind of traffic, but in this case they were spread thinly over 6,000+ ASNs! I ended up blocking all…

I've seen a few attacks where the operators placed malicious code on high-traffic sites (e.g. some government thing, larger newspapers), and then just let browsers load your site as an img. Did you see images, css, js being loaded from these IPs? If they were expecting images, they wouldn't parse the HTML and not load other resources. It's a pretty effective attack because you get large numbers of individual browsers…

I seem to remember a thing china did 10 years back where they injected JavaScript into every web request that went through their Great Firewall to target GitHub… I think it’s known as the “Great Cannon” because they can basically make every Chinese internet user’s browser hit your website in a DoS attack.

Digging it up: https://www.washingtonpost.com/news/the-switch/wp/2015/04/10...

Re: The Web Is Broken – Botnet Part 2

#277

Post by nodogoto »

aorth wrote: post #223

In the last week I've had to deal with two large-scale influxes of traffic on one particular web server in our organization. The first involved requests from 300,000 unique IPs in a span of a few hours. I analyzed them and found that ~250,000 were from Brazil. I'm used to using ASNs to block network ranges sending this kind of traffic, but in this case they were spread thinly over 6,000+ ASNs! I ended up blocking all…

My company's site has also been getting hammered by Brazilian IPs. They're focused on a single filterable table of fewer than 100 rows, querying it with various filter combinations every second of every minute of every day.

Re: The Web Is Broken – Botnet Part 2

#278

Post by klabb3 »

nottorp wrote: post #245
klabb3 wrote: post #243

Earlier quoted context omitted.

> using my bandwidth without my permission What are you talking about? > What does it have to do with p2p? It’s an example of when you design sandboxes/firewalls it’s very easy to assume all apps are one big homogenous blob doing rest calls and everything else is malicious or suspicious. You often need strange permissions to do interesting things. Apple gives themselves these perms all the time.

Wait, why should applications be allowed to do rest calls by default? > What are you talking about? That’s the main use case for p2p in an application isn’t it? Reducing the vendors bandwidth bill…

> That’s the main use case for p2p in an application isn’t it? Reducing the vendors bandwidth bill…

The equivalent would be to say that running local workloads or compute is to reduce the vendors bill. It’s a very centralized view of the internet.

There are many reasons to do p2p. Such as improving bandwidth and latency, circumventing censorship, improve resilience and more. WebRTC is a good example of p2p used by small and large companies alike. None of this is any more ”without permission” than a standard app phoning home and tracking your fingerprint and IP.

Re: The Web Is Broken – Botnet Part 2

#279

Post by spookie »

CalRobert wrote: post #219
hubraumhugo wrote: post #178

We all agree that AI crawlers are a big issue as they don't respect any established best practices, but we rarely talk about the path forward. Scraping has been around for as long as the internet, and it was mostly fine. There are many very legitimate use cases for browser automation and data extraction (I work in this space). So what are potential solutions? We're somehow still stuck with CAPTCHAS, a 25 years old co…

I hate this but I suspect a login-only deanonymised web (made simple with chrome and WEI!) is the future. Firefox users can go to hell.

I'm still surprised by people everyday, after all these years. This is one of those times. Crazy how anyone would ever want a single point of identifying everything you do.

Re: The Web Is Broken – Botnet Part 2

#280

Post by martin82 »

rollcat wrote: post #232
aorth wrote: post #223

In the last week I've had to deal with two large-scale influxes of traffic on one particular web server in our organization. The first involved requests from 300,000 unique IPs in a span of a few hours. I analyzed them and found that ~250,000 were from Brazil. I'm used to using ASNs to block network ranges sending this kind of traffic, but in this case they were spread thinly over 6,000+ ASNs! I ended up blocking all…

Try Anubis: https://anubis.techaro.lol > It's a reverse proxy that presents a PoC challenge to every new visitor. It shifts the initial cost of accessing your server's resources back at the client. Assuming your uplink can handle 300k clients requesting a single 70kb web page, it should solve most of your problems. For science, can you estimate your peak QPS?

This looks very cool, but isn't it just a matter of months until all scrapers get updated and can easily beat this challenge and are able to compute modern JS stuff?
Post reply on HN