Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Aggressive bots ruined my weekend

herman.bearblog.dev

Post reply on HN
Print view View original

91–100 of 109 posts

Re: Aggressive bots ruined my weekend

#91

Post by VladVladikoff »

TYPE_FASTER wrote: post #43
dataviz1000 wrote: post #9

Earlier quoted context omitted.

They provide an SDK for mobile developers. Here is a video of how it works. [0] [0] https://www.youtube.com/watch?v=1a9HLrwvUO4&t=15s

Also see https://www.youtube.com/watch?v=AGaiVApKfmc - "Avoid restrictions and blocks using the fastest and most stable proxy network"...they're pretty upfront with this, aren't they? Oh, and they will sell you the datasets they've already scraped using mobile devices: https://brightdata.com/lp/web-data/datasets This actually explains a phishing attack where I received a text from somebody purporting to be a co-worke…

That scam definitely uses linked in as the source. We get a lot of those BEC emails and it’s always the people who are on LinkedIn. Also keep in mind LinkedIn has had big database leaks in the past, you might not even need to scrape them, just download a huge database from a leaks site.

Re: Aggressive bots ruined my weekend

#92

Post by VladVladikoff »

myaccountonhn wrote: post #15
VladVladikoff wrote: post #4

Earlier quoted context omitted.

This is actually a commonly known fact. There are many services now that sell “residential proxies”, which are always mobile IP addresses. Since mobile IPs use CGNat it’s also not great to block the IP because it can be like geofencing an entire city or town. Some examples are: oxylabs, iproyal, brightdata, etc. Recently I filed an abuse complaint directly with brightdata because I was getting hit with 1000s of reque…

One such example is brightdata, on lobsters someone did a writeup https://lobste.rs/s/pmfuza/bro_ban_me_at_ip_level_if_you_don...

Never heard of lobsters before. Cool site. Seems to be invite only though :( If you could share an invite that would be cool. torosanchez@protonmail.me Thanks!

Re: Aggressive bots ruined my weekend

#93

Post by r_singh »

r_singh wrote: post #73
georgefrowny wrote: post #69

Earlier quoted context omitted.

To be fair the heyday of unshit search was driven by mostly-consensual scraping. Today there are far too many people scraping stuff that isn't intended to be scraped, for profit, and doing it in a heavy-handed way that actually does have a negative and continuous effect on the victim's capacity. Everyone from AI services too lazy or otherwise unwilling to cache to companies exfiltrating some kind of data for their ow…

With peering bandwidth being freely distributed to ISPs and consumers being fed media and subsidised services up until their necks makes the counter argument smell of narrative control rather than technical or financial constraints But as I’m growing older I’m learning that the tech industry is mostly politically driven and relies on truth obfuscation as explained by Peter Thiel rather than real empowerment It’s faci…

I may have been too harsh. I love capitalism, technology, and software—they’ve built a meritocratic world and given me the tools to build my own life.

AI and technology feel like my best friend, but also my worst enemy when they edge toward learned helplessness. That tension exists with anything we depend on: the closer we get, the more power it holds.

The relationship between user and technology is becoming deeply intimate as systems gain reach and control. It’s important to stay optimistic but skeptical—and to keep protesting everything—because the work is moving faster than our ability to register its consequences.

Reading back, I realise I drifted into more of a monologue than a conversation. I get carried away when I’m trying to reason things out in public. Still, I stand by the core point about balance and transparency in how we shape the web.

Re: Aggressive bots ruined my weekend

#94

Post by wiredfool »

karlshea wrote: post #48
Neil44 wrote: post #23

Earlier quoted context omitted.

Same, I have a few hundred Wordpress sites and bot activity has ramped up a lot over the last year or two. AI scrapers can be quite aggressive and often generate a ton of requests where for example a site has a lot of parameters, the bot will go nuts seeming to iterate through all possible parameters. Sometimes I dig in and try to think of new rules to block the bulk, but I am also wary of AI replacing Google and not…

A client of mine had this exact problem with faceted search, and putting the site behind Fastly didn’t help since you can’t cache millions of combinations. And they don’t have the budget for more than one origin server. The solution was if you’ve got “bot” in your UA Fastly’s VCL returns a 403 with any facet query param. Problem solved. And it’s not going to break anything, all of the information is still accessible…

I see a ton of random recent semi reasonable user agents now, and some of them are even sending the sec-ua, reasonable accept headers and the more obscure headers.

Re: Aggressive bots ruined my weekend

#95

Post by LilBytes »

uvaursi wrote: post #45
reustle wrote: post #37

Earlier quoted context omitted.

Cloudflare is not a solution. Only leading to a further centralized internet.

I think the OP is suggesting using a caching layer at HTTP output, and suggesting CF as an option (a quick/cheap one). If you have an axe to grind with CF you can take it up with them, but it’s an option. Feel free to suggest others.

It seems to me that the writer, Herman, doesn't want to use the cloud short of the parts that are almost mandatory these days (e.g., CloudFlare, or a CDN of some sort).

GitHub Pages, CloudFlare pages etc are a great and very simple service. But they're opposite or contrary to running your own hardware, warts and all.

Herman wasn't looking for solutions IMO, I read it more as him lamenting at how hostile and insidious the internet has become. It has been for some time, but it seems to be getting exponentially worse.

Re: Aggressive bots ruined my weekend

#96

Post by karlshea »

r_singh wrote: post #59
karlshea wrote: post #40

Earlier quoted context omitted.

There were already guidelines, these trash people aren’t following them. That’s why there’s now “sentiment” against them.

It’s fair to be angry at abuse and "aggressive bots", but it's important to remember most large platforms—including the ones being scraped—built their own products on scraping too. I run an e-commerce-specific scraping API that helps developers access SERP, PDP, and reviews data. I've noticed the web already has unsaid balances: certain traffic patterns and techniques are tolerated, others clearly aren’t. Most sites…

If you’re running a well-behaved crawler (for example one that respects nofollow, and doesn’t try every single product filter combination it can find) then fine. If you don’t, then I don’t have any sympathy for the consequences that your niche of the industry caused.

Not everyone has the budget for unlimited bandwidth and compute, and in several of my clients’ cases that’s been >95% of all traffic.

People running these bots with AI/VC capital are just script kiddies that forgot that not every site is a boatload of app servers behind Cloudflare.

Re: Aggressive bots ruined my weekend

#97

Post by r_singh »

karlshea wrote: post #96
r_singh wrote: post #59

Earlier quoted context omitted.

It’s fair to be angry at abuse and "aggressive bots", but it's important to remember most large platforms—including the ones being scraped—built their own products on scraping too. I run an e-commerce-specific scraping API that helps developers access SERP, PDP, and reviews data. I've noticed the web already has unsaid balances: certain traffic patterns and techniques are tolerated, others clearly aren’t. Most sites…

If you’re running a well-behaved crawler (for example one that respects nofollow, and doesn’t try every single product filter combination it can find) then fine. If you don’t, then I don’t have any sympathy for the consequences that your niche of the industry caused. Not everyone has the budget for unlimited bandwidth and compute, and in several of my clients’ cases that’s been >95% of all traffic. People running the…

My service only extracts public data major retailers, not indie sites, and deducts more credits for lower-traffic domains to offset load differences.

It would be great if there were reliable ways to distinguish good bots from bad ones — many actually improve discoverability and sales. I see this with affiliate shopping sites that depend on e-commerce data, though that impact is hard to trace directly.

The bad actors are the ones cloning sites or using data for manipulation and propaganda.

Re: Aggressive bots ruined my weekend

#98

Post by josephb »

TimorousBestie wrote: post #41
flaviuspopan wrote: post #35

Earlier quoted context omitted.

His persistent efforts are the reason I pay for Bear Blog. I think he should fight for the chance to come out on the other side of whatever future we’re heading towards.

I pay for Bear Blog, too. But this year has been problem after problem for its sole proprietor, and I don’t think it’s going to get better.

As much as I love the service and enjoy the transparency, I do wonder about the future of smaller operator services as the Internet continues it's descent into a giant mess :-)

Re: Aggressive bots ruined my weekend

#99

Post by [deleted] »

VladVladikoff wrote: post #90
cuu508 wrote: post #44

Earlier quoted context omitted.

IMO Google Play should check apps for presence of this SDK and other similar SDKs, and, upon detection, treat these apps as malware.

I was wondering if they already do but maybe it’s a cat and mouse game where those companies obfuscate their code to avoid automated detection.

[deleted]

Re: Aggressive bots ruined my weekend

#100

Post by uvaursi »

LilBytes wrote: post #95
uvaursi wrote: post #45

Earlier quoted context omitted.

I think the OP is suggesting using a caching layer at HTTP output, and suggesting CF as an option (a quick/cheap one). If you have an axe to grind with CF you can take it up with them, but it’s an option. Feel free to suggest others.

It seems to me that the writer, Herman, doesn't want to use the cloud short of the parts that are almost mandatory these days (e.g., CloudFlare, or a CDN of some sort). GitHub Pages, CloudFlare pages etc are a great and very simple service. But they're opposite or contrary to running your own hardware, warts and all. Herman wasn't looking for solutions IMO, I read it more as him lamenting at how hostile and insidious…

Remember when Stormfront was a thing? Remember how everyone cheered on that CF and others banned them? I’m sure Herman was one of the loudest ones to proclaim victory that day.

Hostility and insidiousness were created by you for not standing up when it was most needed and called for. And as you can see, Stormfront in hindsight was the most milquetoast website compared to landscape of politics today. All I’m saying is CF is a viable caching option. But if you’re looking for morality support - you lost that war a long time ago.

“If we do not stand up for the rights of the accused, we endanger our own. For when the tide turns, who will stand up for us?”

Post reply on HN