201–210 of 230 posts
Note that even after the disruption, I'm still able to route millions of requests/day through IP IDEA's network
Earlier quoted context omitted.
One thing about Google is that many anti-scraping services explicitly allow access to Google and maybe couple of other search engines. Everybody else gets to enjoy CloudFlare captcha, even when doing crawling at reasonable speeds. Rules For Thee but Not for Me
Why are you scraping sites in the first place? What legitimate reason is there for you doing that?
Do you think this is such a horrible thing to scrape? I can't do it manually since there are few hundred locations. I could write some python script which uses playwrite to scrape things using my desktop browser in order to avoid CloudFlare. Or, which I am much more familiar with, I could write a python script that uses BeautifulSoup to extract all the relevant locations once for me. I would have been perfectly happy fetching 1 page/sec or even 1 page/2 seconds and would still be done within 20 minutes if only there was no anti-scraping protection.
Scraping is a perfectly legal activity, after all. Except thanks to overly-eager scraping bots and clueless/malicious people who run them there's very little chance for anyone trying to compete with Google or even do small scale scraping to make their life and life of local art enthusiasts easier. Google owns search. Google IS search and no competition is allowed, it seems.
Earlier quoted context omitted.
First of: Google has not once crashed one of our sites with GoogleBot. They have never tried to by-pass our caching and they are open and honest about their IP ranges, allowing us to rate-limit if needed. The residential proxies are not needed, if you behave. My take is that you want to scrape stuff that site owners do not want to give you and you don't want to be told no or perhaps pay a license. That is the only ca…
You can’t get much crawling done from published cloud IPs. Residential proxies are the only way to do most crawls today. That said, I support Google working to shut these networks down, since they are almost universally bad. It’s just a shame that there’s no where to go for legitimate crawling activities.
Think about why that might be. I'm sorry, if you legitimately need to crawl the net, and do so from a cloud provide, your industry screwed you over with bad behaviour. Go get hosting with a company that cares about who their customers are, you're hanging out with a bad crowd.
Earlier quoted context omitted.
You can’t get much crawling done from published cloud IPs. Residential proxies are the only way to do most crawls today. That said, I support Google working to shut these networks down, since they are almost universally bad. It’s just a shame that there’s no where to go for legitimate crawling activities.
> You can’t get much crawling done from published cloud IPs. Think about why that might be. I'm sorry, if you legitimately need to crawl the net, and do so from a cloud provide, your industry screwed you over with bad behaviour. Go get hosting with a company that cares about who their customers are, you're hanging out with a bad crowd.
My understanding is that routing through residential IPs is a part of the business of some VPN providers. I don't know how above board they are on this (as in notifying customers that this may happen, however buried in the usage agreement, or even allowing them to opt out). But, my main point, is that the whole business is "on the up and up" vs some dark botnet.
Saying you don't know something in the comments of an article that explains that thing is a bold strategy
Earlier quoted context omitted.
I'd still like the ability to just block a crawler by its IP range, but these days nope. 1 Hz is 86400 hits per day, or 600k hits per week. That's just one crawler. Just checked my access log... 958k hits in a week from 622k unique addresses. 95% is fetching random links from u-boot repository that I host, which is completely random. I blocked all of the GCP/AWS/Alibaba and of course Azure cloud IP ranges. It's almos…
I’ve been enduring that exact same traffic pattern. I used Anubis and a cookie redirect to cut the load on my Forgejo server by around 3 orders of magnitude: https://honeypot.net/2025/12/22/i-read-yann-espositos-blog.h...
I guess the bots are all spoofing consumer browser UAs and just the slightest friction outside of well-known tooling will deter them completely.
Earlier quoted context omitted.
> You can’t get much crawling done from published cloud IPs. Think about why that might be. I'm sorry, if you legitimately need to crawl the net, and do so from a cloud provide, your industry screwed you over with bad behaviour. Go get hosting with a company that cares about who their customers are, you're hanging out with a bad crowd.
what industry is that? Every industry is on the cloud.
This differs obviously, but having an ASN in our case means that we can deal you, contact you and assume that you're better than random bot number 817.
My understanding is that routing through residential IPs is a part of the business of some VPN providers. I don't know how above board they are on this (as in notifying customers that this may happen, however buried in the usage agreement, or even allowing them to opt out). But, my main point, is that the whole business is "on the up and up" vs some dark botnet.
[1] Using the website mentioned by user Rasbora https://news.ycombinator.com/item?id=46837806
I'm surprised by the negative takes... Yes, proxies are good. Ones which you pay for and which are running legitimately, with the knowledge (and compensation) of those who run them. Malware in random apps running on your device without your knowledge is bad.
The problem is, it is by default unethical to have residential users be exit nodes for VPNs - unless these users are lawyers or technical experts.
No matter what you do as a "residential proxy" company - you cannot prevent your service being used by CSAM peddlers, and thus you cannot prevent that your exit nodes aren't the ones whose IP addresses show up when the FBI comes knocking.