Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Aggressive bots ruined my weekend

herman.bearblog.dev

Post reply on HN
Print view View original

71–80 of 109 posts

Re: Aggressive bots ruined my weekend

#71

Post by Cthulhu_ »

Zanfa wrote: post #26
asplake wrote: post #3

> What's wild is that these scrapers rotate through thousands of IP addresses during their scrapes, which leads me to suspect that the requests are being tunnelled through apps on mobile devices, since the ASNs tend to be cellular networks. I'm still speculating here, but I think app developers have found another way to monetise their apps by offering them for free, and selling tunnel access to scrapers. Wild indeed,…

SIM farms are another possible explanation. FBI just busted one with hundreds of thousands of SIMs just a few weeks ago.

Wouldn't the network providers be able to detect those? I'm fairly sure they don't like their networks being abused either... or they don't really care because they get paid per connection.

edit: Actually this is what I'm getting increasingly angry about: providers and platforms not doing anything against bots or low value stuff (think Amazon dropshippers too) because any usage of their service, bots or otherwise, are metrics going up and metrics going brrt means profit and shareholder interest.

Re: Aggressive bots ruined my weekend

#72

Post by nerdponx »

TYPE_FASTER wrote: post #43
dataviz1000 wrote: post #9

Earlier quoted context omitted.

They provide an SDK for mobile developers. Here is a video of how it works. [0] [0] https://www.youtube.com/watch?v=1a9HLrwvUO4&t=15s

Also see https://www.youtube.com/watch?v=AGaiVApKfmc - "Avoid restrictions and blocks using the fastest and most stable proxy network"...they're pretty upfront with this, aren't they? Oh, and they will sell you the datasets they've already scraped using mobile devices: https://brightdata.com/lp/web-data/datasets This actually explains a phishing attack where I received a text from somebody purporting to be a co-worke…

It should be illegal, but this stuff is propping up the appearance of a healthy economy so nobody will touch it.

Re: Aggressive bots ruined my weekend

#73

Post by r_singh »

georgefrowny wrote: post #69
intended wrote: post #39

Earlier quoted context omitted.

What ? What do you mean ?

To be fair the heyday of unshit search was driven by mostly-consensual scraping. Today there are far too many people scraping stuff that isn't intended to be scraped, for profit, and doing it in a heavy-handed way that actually does have a negative and continuous effect on the victim's capacity. Everyone from AI services too lazy or otherwise unwilling to cache to companies exfiltrating some kind of data for their ow…

With peering bandwidth being freely distributed to ISPs and consumers being fed media and subsidised services up until their necks makes the counter argument smell of narrative control rather than technical or financial constraints

But as I’m growing older I’m learning that the tech industry is mostly politically driven and relies on truth obfuscation as explained by Peter Thiel rather than real empowerment

It’s facilitating accumulation of control and power at an unparalleled pace. If anything it’s proving to be more unjust than the feudal systems it promises to replace.

Re: Aggressive bots ruined my weekend

#74

Post by Cthulhu_ »

uvaursi wrote: post #64
Chabsff wrote: post #52

Earlier quoted context omitted.

The bots flock to where the data is. Moving to a different network is just begging for the bots to tag along.

We can set different rules on these networks however. We can choose to be choosy at the gate.

How though? Bots evade bot detection mechanisms, as described in other threads. Unless you introduce something like ID verification or pay-per-request making bot traffic too expensive for the bots. But these techniques have been posited for older generations of bot traffic too.

Re: Aggressive bots ruined my weekend

#75

Post by Cthulhu_ »

r_singh wrote: post #34

The Internet isn’t possible without scraping. For all the sentiment against scraping public data, doing so remains legal and essential to a lot of the services we use everyday. I think setting guidelines and shaping the web for reduced friction aimed at fair usage rather than turning it political would be the right thing to do.

Well sure, but these guidelines exist, the robots.txt guidelines has been an industry-led, self-governing / self-restrictive standard. But newer bots ignore them. It'll take years for legislation to catch up, and even then it would be by country or region, not something global because that's not how the internet works.

Even if there is legislation or whatever, you can sue an OpenAI or a Microsoft, but starting a new company that does scraping and sells it on to the highest bidder is trivial.

Re: Aggressive bots ruined my weekend

#76

Post by codersfocus »

pixl97 wrote: post #57
FeepingCreature wrote: post #14

Earlier quoted context omitted.

In principle, it should be possible to identify malign IPs at scale by using a central service and reporting IPs probabilistically. That is, if you report every thousandth page hit with a simple UDP packet, the central tracker gets very low load and still enough data to publish a bloom filter of abusive IPs, say a million bits gives you pretty low false-positive. (If it's only ~10k malign IPs, tbh you can just keep a…

>malign IPs at scale As talked about elsewhere in this thread, residential devices being used as proxies behind CGNAT ruins this. Not getting rid of IPv4 years ago is finally coming to bite us in the ass in a big way.

IPv6 wouldn't solve this, since IPs would be too cheap to meter.

Re: Aggressive bots ruined my weekend

#77

Post by embedding-shape »

Cthulhu_ wrote: post #74
uvaursi wrote: post #64

Earlier quoted context omitted.

We can set different rules on these networks however. We can choose to be choosy at the gate.

How though? Bots evade bot detection mechanisms, as described in other threads. Unless you introduce something like ID verification or pay-per-request making bot traffic too expensive for the bots. But these techniques have been posited for older generations of bot traffic too.

Make it P2P and content-based, instead of location-based like the current web. Content could be served from anywhere, so DDoS stops being an effective method, and shared peer quality could propagate across the network to ban bad actors quickly.

I spend about 30 seconds thinking about this, so this is clearly the perfect solution with zero drawbacks or tradeoffs.

Re: Aggressive bots ruined my weekend

#78

Post by r_singh »

Cthulhu_ wrote: post #75
r_singh wrote: post #34

The Internet isn’t possible without scraping. For all the sentiment against scraping public data, doing so remains legal and essential to a lot of the services we use everyday. I think setting guidelines and shaping the web for reduced friction aimed at fair usage rather than turning it political would be the right thing to do.

Well sure, but these guidelines exist, the robots.txt guidelines has been an industry-led, self-governing / self-restrictive standard. But newer bots ignore them. It'll take years for legislation to catch up, and even then it would be by country or region, not something global because that's not how the internet works. Even if there is legislation or whatever, you can sue an OpenAI or a Microsoft, but starting a new…

As the legal history around scraping shows, it’s almost always the smaller company that gets sued out of existence. Taking on OpenAI or Microsoft, as you suggest, isn’t realistic — even governments often struggle to hold them accountable.

And for the record, large companies regularly ignore robots.txt themselves: LinkedIn, Google, OpenAI, and plenty of others.

The reality is that it’s the big players who behave like the aggressors, shaping the rules and breaking them when convenient. Smaller developers aren’t the problem, they’re just easier to punish.

Re: Aggressive bots ruined my weekend

#79

Post by yomismoaqui »

nurettin wrote: post #24
corbet wrote: post #18

Earlier quoted context omitted.

The "compliance officer" at Bright Data, instead, offered me a special deal to protect my site from their bots ... they run a protection racket along with all the rest of their nastiness.

I worked for an Amazon scraping business and they used Luminati (Now Brightdata) for a few months until I figured out a way to avoid the ban hammer and got rid of their proxy. They indeed provided "high quality" residential and cellular ips and "normal quality" data center ips. You had to keep cycling the ip pool every 2-3 days which cost extra. It felt super shady. It isn't their bots, they lease connections to whoe…

> ... until I figured out a way to avoid the ban hammer ...

You had my curiosity ... but now you have my attention.

Re: Aggressive bots ruined my weekend

#80

Post by Chabsff »

embedding-shape wrote: post #77
Cthulhu_ wrote: post #74

Earlier quoted context omitted.

How though? Bots evade bot detection mechanisms, as described in other threads. Unless you introduce something like ID verification or pay-per-request making bot traffic too expensive for the bots. But these techniques have been posited for older generations of bot traffic too.

Make it P2P and content-based, instead of location-based like the current web. Content could be served from anywhere, so DDoS stops being an effective method, and shared peer quality could propagate across the network to ban bad actors quickly. I spend about 30 seconds thinking about this, so this is clearly the perfect solution with zero drawbacks or tradeoffs.

I know this is tongue in cheek, but I'll give it a serious reply in case someone finds themselves "inspired".

A CDN. What you are describing is a CDN. We have CDNs today and the problem still exists because most of today's websites refuse to operate within the constraints. There is no need for new infrastructure to deploy this solution, we just need website operators to "give up" and operate in a more static way.

Post reply on HN