Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Aggressive bots ruined my weekend

herman.bearblog.dev

Post reply on HN
Print view View original

51–60 of 109 posts

Re: Aggressive bots ruined my weekend

#51

Post by psnehanshu »

ItsBob wrote: post #22

I had a website earlier this year running on Hetzner. It was purely experimenting with some ASP.NET stuff but when looking at the logs, I noticed a shit-load of attempts at various WordPress-related endpoints. I then read something about a guy who deliberately put a honeypot in his robots.txt file. It was pointing to a completely bogus endpoint. Now, the theory was, humans won't read robots.txt so there's no danger,…

What if it was proxied through mobile network on an unsuspecting user's phone? You risk of blocking a whole city or region.

Re: Aggressive bots ruined my weekend

#52

Post by Chabsff »

uvaursi wrote: post #49

Do we shift over everything to le Dark Web and let the corpos use this one for selling their shit to consumers? These toys don’t want to play nice and there’s no real way to stop them without bringing in things like Real ID and other verifications that infringe on anonymity.

The bots flock to where the data is. Moving to a different network is just begging for the bots to tag along.

Re: Aggressive bots ruined my weekend

#53

Post by dylan604 »

nemomarx wrote: post #33
wat10000 wrote: post #21

Earlier quoted context omitted.

Lately Reddit has been showing me posts in subreddits for some of these services. They pitch "passive income" by sharing your connection, an easy way to make a few bucks by renting out your unused capacity. What happens is that you become an endpoint for their shady VPNs. These subreddits are full of people complaining that they're getting hit by abuse complaints from their ISPs. Naturally, these services claim to fo…

Salad, right? What a strange business

Why is it strange. Of course it exists.

Re: Aggressive bots ruined my weekend

#54

Post by cupofjoakim »

2OEH8eoCRo0 wrote: post #16

Why don't we sue the abusive scrapers? Scraping is legal but DDoSing is not!

Not sure if that's satire or not but how would you even identify the party to sue? What do you do if they're based in a country where you can't sue them ofer relatively trivial matters as this?

Re: Aggressive bots ruined my weekend

#55

Post by y-zon128 »

> Auto-restart the reverse-proxy if bandwidth usage drops to zero for more than 2 minutes

It's understandable in your case as you have traffic coming in constantly, but first thing that came to my mind is a loop of contant reboots - again, very unlikely in your case. Sometimes such blanket rules hit me due to most unexpected reasons, like the proxy somehow failed to start serving traffic in the given timeframe.

Though I completely appreciate and agree with the 'ship now something that works now' approach!

Re: Aggressive bots ruined my weekend

#56

Post by 2OEH8eoCRo0 »

cupofjoakim wrote: post #54
2OEH8eoCRo0 wrote: post #16

Why don't we sue the abusive scrapers? Scraping is legal but DDoSing is not!

Not sure if that's satire or not but how would you even identify the party to sue? What do you do if they're based in a country where you can't sue them ofer relatively trivial matters as this?

Not satire but it's a huge problem with the internet. Everyone washes their hands and people can harm you without liability.

Re: Aggressive bots ruined my weekend

#57

Post by pixl97 »

FeepingCreature wrote: post #14
cupofjoakim wrote: post #13

We feel this at work too. We run a book streaming platform with all books, booklists, authors, narrators and publishers available as standalone web pages for SEO, in the multiple millions. Last 6 months have turned into a hellscape - for a few reasons: 1. It's become commonplace to not respect rate limits 2. Bots no longer identify themselves by UA 3. Bots use VPNs or similar tech to bypass ip rate limiting 4. Bots u…

In principle, it should be possible to identify malign IPs at scale by using a central service and reporting IPs probabilistically. That is, if you report every thousandth page hit with a simple UDP packet, the central tracker gets very low load and still enough data to publish a bloom filter of abusive IPs, say a million bits gives you pretty low false-positive. (If it's only ~10k malign IPs, tbh you can just keep a…

>malign IPs at scale

As talked about elsewhere in this thread, residential devices being used as proxies behind CGNAT ruins this. Not getting rid of IPv4 years ago is finally coming to bite us in the ass in a big way.

Re: Aggressive bots ruined my weekend

#58

Post by pjc50 »

immibis wrote: post #27
asplake wrote: post #3

> What's wild is that these scrapers rotate through thousands of IP addresses during their scrapes, which leads me to suspect that the requests are being tunnelled through apps on mobile devices, since the ASNs tend to be cellular networks. I'm still speculating here, but I think app developers have found another way to monetise their apps by offering them for free, and selling tunnel access to scrapers. Wild indeed,…

You can get paid a few dollars (not many) to let them use your connection. I would like Cloudflare's business model (blocking datacenter IPs) to be worthless, so I do it. Haven't tried a withdrawal yet so it could well be a scam. This is not illegal (unless it's a scam).

This is one of those "ACAB" things where you might reasonably dislike Cloudflare but a world without them or an equivalent will evolve worse solutions to the same problems, which you will like even less.

Re: Aggressive bots ruined my weekend

#59

Post by r_singh »

karlshea wrote: post #40
r_singh wrote: post #34

The Internet isn’t possible without scraping. For all the sentiment against scraping public data, doing so remains legal and essential to a lot of the services we use everyday. I think setting guidelines and shaping the web for reduced friction aimed at fair usage rather than turning it political would be the right thing to do.

There were already guidelines, these trash people aren’t following them. That’s why there’s now “sentiment” against them.

It’s fair to be angry at abuse and "aggressive bots", but it's important to remember most large platforms—including the ones being scraped—built their own products on scraping too.

I run an e-commerce-specific scraping API that helps developers access SERP, PDP, and reviews data. I've noticed the web already has unsaid balances: certain traffic patterns and techniques are tolerated, others clearly aren’t. Most sites handle reasonable, well-behaved crawlers just fine.

Platforms claim ownership of UGC and public data through dark patterns and narrative control. The current guidelines are a result of supplier convenience, and there are several cases where absolutely fundamental web services run by the largest companies in the world themselves breach those guidelines (including those funded by the fund running this site). We need standards that treat public data as a shared resource with predictable, ethical access for everyone, not just for those with scale or lobbying power.

Re: Aggressive bots ruined my weekend

#60

Post by arethuza »

VladVladikoff wrote: post #10
dataviz1000 wrote: post #9

Earlier quoted context omitted.

They provide an SDK for mobile developers. Here is a video of how it works. [0] [0] https://www.youtube.com/watch?v=1a9HLrwvUO4&t=15s

WOW that video! Ain’t no way anyone has EVER read those terms. This feels so insidious that it really should be illegal. Wonder if this exists in the EU or if they have shut it down already?

That video has the app asking the user to confirm the use of their device to run a proxy within the app - but is there any hard requirement for this, could apps use this SDK and silently run as a proxy?
Post reply on HN