Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gentoo bugzilla closed due AI bot scraper overload

social.treehouse.systems

Post reply on HN
Print view View original

111–120 of 125 posts

Re: Gentoo bugzilla closed due AI bot scraper overload

#111

Post by sandeepkd »

jancsika wrote: post #102
sandeepkd wrote: post #86

Earlier quoted context omitted.

You need to account for the fact that a lot of scraping is delegated by the big players to smaller players who can take loss of reputation and can use questionable methods (residential IPs). Some of the scrapers from these AI companies have been written very poorly from performance perspective. From what I understand the pressure has created even Google to be a lot more aggressive than what it was before. Not entirel…

Still not sure about the "who" question, but here's the scale of money changing hands: > Residential proxies are everywhere, so why did proxy DDoS attacks mostly come from the U.S.? The answer is money. If you are committing fraud or circumventing content restrictions, a Russian IP address gets geo-blocked instantly. A fresh U.S. residential IP address (especially one behind carrier-grade NAT and harder to block indi…

Here is the fun part, the same companies invests millions of dollars to protect its infrastructure and services from scrapers/bots AND they also invest millions of dollars to circumvent the bots, captchas and rate limits.

Unfortunately you cant have it both ways which makes it hard problem to solve for everyone

Re: Gentoo bugzilla closed due AI bot scraper overload

#112

Post by rufasterisco »

Certhas wrote: post #82
rufasterisco wrote: post #67

Enough with playing around the issue. The way out of this mess is not to protect with tech that works but with principles and laws. This is not a tech problem. This is about what should or should not be legal. Nor it’s a question of having time to implement solution X or Y. If someone attacks you yes, you should have better security but you also need to have legal recourse, or it will never stop. Ddos is already ille…

I agree in principle. The tech community naturally has a tendency to reach for tech solutions to societal problems. But without jurisdiction, laws are meaningless. If you create a law that says you have to honour robots.txt files, what do you do if an IP from another country fails to do so?

You check the laws of your country. Likely they say that DDOS are criminal offences, and you can get 10 or 20 years of jail time, already.

Then you report a crime and send the logs. And maybe report to some politician that is tough on crime. And you are not alone.

I am sure you follow through, Pls check also my other comments.

Laws without enforcement are meaningless, without jurisdiction they are less effective.

Maybe yes, some poor guy will find a letter that says he needs to pay 50 USD or EUR to some faraway country because the puzzle app in his tv has weird ToS. Or whatever, a cease and desist letter? A “you are part of a hacking circle please uninstall this app” letter, signed by a judge?

I don’t advocate for people getting thrown in jail, I just think that the whole “residential proxies” issue might be a charade that would not last longer if citizens and businesses would claim rights instead of play victim.

I also really dislike grasping for a tech solution because I don’t want to start to mine crypto to read an article or to depend on some “free” cloud service to host an open bug tracker.

It just feels so weird: why is it that no-one is doing it? I feel like I an one “comment away” from someone explaining to me how naive I am, and yet it’s never coming.

Re: Gentoo bugzilla closed due AI bot scraper overload

#113

Post by simmerup »

userbinator wrote: post #107

Manufactured consent. This is not AI, this is just DDoS'ing for extortion and driving towards the closed proprietary "can only be accessed by a sanctioned UA" Internet that they're after. Why else do you think identity verification and attempting to turn the Internet into a walled garden became such a big thing in the recent years?

You don’t think it has anything to do with hostile states using fake profiles on social media to drum up discontent in the democratic world?

Re: Gentoo bugzilla closed due AI bot scraper overload

#114

Post by rufasterisco »

xena wrote: post #68
rufasterisco wrote: post #60

Earlier quoted context omitted.

Instead of punishing a criminal, you make everybody pay. This is not only unfair to legitimate users having to pay. It also needs to explain what actually happens when the Ddos succeeds (I know, you are talking about scrapers, but what’s the difference really?). Does that mean that the attacker just gets to shrug it off. Don’t take my statements as facts, I just want to outline a few reason I could come up with that…

If we could identify who is doing this, I bet that those people would be publicly burned at the stake by now.

I don’t like people being burnt at stakes and what I see when I google “residential proxy” does not feel like we cannot identify. More like we are not even trying.

I mean, if residential proxies are tools used to DDOS websites, why is Google advertising them? And if you have Google ads for them, can we really say “ah, if we only knew who these guys are?”.

Re: Gentoo bugzilla closed due AI bot scraper overload

#115

Post by rcxdude »

rufasterisco wrote: post #110
rcxdude wrote: post #85

Earlier quoted context omitted.

The question is, even if you identify who is the source of the traffic, are they even in a place where you can realistically sue them? The problematic traffic generally is not the bots that identify themselves, but the ones that are using residential proxies and try to be a non-fingerprintable as possible.

Then enforcement will target companies providing residential proxy services. In whatever country they are. I am not saying this is the magic bullet that will make it all go away. Some will, some will not. Enforcement’s aim is not to completely eradicate crime: make it anti-economical for the majority and it’s a good deal enough. Go to Berlin and torrent a movie: you will be fined, for a few hundred euros. Can you VPN…

The residential proxy operators are also not often within easy reach of western justice systems. Fines for torrenting movies are targeted against end users within the countries (and also not exactly effective, either, because the issue is quite diffuse), but prosecuting criminals in other countries, especially countries which don't really care to stop those criminals so long as they are causing problems for other countries, is not an easy process. The internet is an global system but criminal law generally is not.

Re: Gentoo bugzilla closed due AI bot scraper overload

#116

Post by br0ceph »

jancsika wrote: post #102
sandeepkd wrote: post #86

Earlier quoted context omitted.

You need to account for the fact that a lot of scraping is delegated by the big players to smaller players who can take loss of reputation and can use questionable methods (residential IPs). Some of the scrapers from these AI companies have been written very poorly from performance perspective. From what I understand the pressure has created even Google to be a lot more aggressive than what it was before. Not entirel…

Still not sure about the "who" question, but here's the scale of money changing hands: > Residential proxies are everywhere, so why did proxy DDoS attacks mostly come from the U.S.? The answer is money. If you are committing fraud or circumventing content restrictions, a Russian IP address gets geo-blocked instantly. A fresh U.S. residential IP address (especially one behind carrier-grade NAT and harder to block indi…

if they pay 190/mo to proxy, i should just sell my ip direct -> profit i pay less than half that for fiber , should get a few more isp accounts, and rake it in.

Re: Gentoo bugzilla closed due AI bot scraper overload

#117

Post by eric__cartman »

pduggishetti wrote: post #25
csomar wrote: post #21

Earlier quoted context omitted.

How many pages does the average software developer visit everyday? 1.000? Price at 0.001 per load and it’ll be completely impractical for crawlers but super cheap for the average connected human on earth.

That would gate the internet to whole lower income countries

Also a great way to block the access to people that do not have a payment system available. Whether it is because they are not in a country that's well integrated into an international payment system, or because they are under 18.

I browsed a lot of those sites when I was a teenager and did not have a credit card or easy access to internet money.

Re: Gentoo bugzilla closed due AI bot scraper overload

#118

Post by csomar »

pduggishetti wrote: post #25
csomar wrote: post #21

Earlier quoted context omitted.

How many pages does the average software developer visit everyday? 1.000? Price at 0.001 per load and it’ll be completely impractical for crawlers but super cheap for the average connected human on earth.

That would gate the internet to whole lower income countries

Only very minimally. A dollar is no longer worth that much. People who live under 1-2 dollars a day probably aren't browsing that much internet per day.

- https://ourworldindata.org/grapher/share-living-with-less-th...

Re: Gentoo bugzilla closed due AI bot scraper overload

#119

Post by csomar »

tredre3 wrote: post #70
bluefirebrand wrote: post #31

Earlier quoted context omitted.

Aren't many of those countries basically being de-facto blocked by Cloudflare filtering out spam already anyways? Genuine question, I'm not up to date on how Cloudflare operates right now

The genuine answer to your question is no, they are not de-facto blocked, unless the website operator chooses to do so.

It's the default setting and most website operators have no idea what they are doing.

Re: Gentoo bugzilla closed due AI bot scraper overload

#120

Post by marginalia_nu »

rcxdude wrote: post #115
rufasterisco wrote: post #110

Earlier quoted context omitted.

Then enforcement will target companies providing residential proxy services. In whatever country they are. I am not saying this is the magic bullet that will make it all go away. Some will, some will not. Enforcement’s aim is not to completely eradicate crime: make it anti-economical for the majority and it’s a good deal enough. Go to Berlin and torrent a movie: you will be fined, for a few hundred euros. Can you VPN…

The residential proxy operators are also not often within easy reach of western justice systems. Fines for torrenting movies are targeted against end users within the countries (and also not exactly effective, either, because the issue is quite diffuse), but prosecuting criminals in other countries, especially countries which don't really care to stop those criminals so long as they are causing problems for other cou…

Yeah, countries are even using this as a bargaining chip, if not a form of hybrid warfare. Neither Russia nor China is losing any sleep over digital attrition in the west.
Post reply on HN