Another idea: Buying an LTE modem/router. That IP address would probably be harder to block.
Cloudproxy – hide your scrapers IP behind the cloud
21–30 of 116 posts
Re: Cloudproxy – hide your scrapers IP behind the cloud
#22Is it ethical to scrape things, trying to bypass prevention methods, against the expressed wishes of a site?
> Is it ethical to scrape things, trying to bypass prevention methods, against the expressed wishes of a site?
No, but many sites only allow Google to index them now.
I don't think that was the original plan for the web.
Re: Cloudproxy – hide your scrapers IP behind the cloud
#23Earlier quoted context omitted.
Are there even ethical or “official” ways to source residential IPs? I thought most of them were from botnet infected IoT devices.
I've heard of services that offer free proxy bandwidth in exchange for the user acting as a residential proxy node for other users and/or paying clients. It's generally marketed as a way for the user to avoid geoblocks and such. If this is clearly stated to the user upfront rather than buried behind half a dozen dark patterns and fine print, it seems like this business model could be conducted ethically. When it come…
You have autobuying bots (sneakers or GPUs or other limited edition stuff) that need to do tricks like this. I wouldn't necessarily call them unethical.
Re: Cloudproxy – hide your scrapers IP behind the cloud
#24Earlier quoted context omitted.
I heard a while back, on HN 8 I think, that a lot of residential proxies are computers that have been infected with malware. How true is that?
There have been a number of Chrome extensions which monetize by selling your bandwidth to provide this kind of service. Not ethical but probably not a compromise.
Browser extensions, and VPNs. Notably, Luminati was allegedly selling HolaVPN users as exit nodes. https://www.trendmicro.com/vinfo/hk-en/security/news/cybercr...
Re: Cloudproxy – hide your scrapers IP behind the cloud
#25In a related note the US Supreme Court just 10 days ago vacated a previous ruling against LinkedIn blocking a Scraper service. So the issue is back to the 9th circuit for a new determination that might, if reversed, change the landscape for Cloudflare and anyone else these operators try to sit behind.
https://www.scotusblog.com/case-files/cases/linkedin-corp-v-...
Re: Cloudproxy – hide your scrapers IP behind the cloud
#26Hiding behind any larger cloud or VPN looks futile. If there's one place to hide scraper nowadays it's Apple's iCloud+ or whatever stupid name they've picked - two proxies to mimic tor.
It’d be easier to use Cloudflare WARP command line client for the same purpose. Apple uses Cloudflare for their private relay so there’s a chance that these IPs won’t be blocked.
If I remember correctly, Cloudflare WARP forwards your client IP in the header. So it probably won't be useful for scraping.
Re: Cloudproxy – hide your scrapers IP behind the cloud
#27Even though I only do it for hobby projects, crawling pages is becoming increasingly difficult unless you are a big player like Google or Microsoft with a whitelisted IP range.
I've had some success in scraping lately with a similar project called FlareSolverr(1).
It's purpose it to get you access to sites which won't let you crawl unless you are using a real browser (e.g amazon, instagram). It doesn't hide your IP but uses puppeteer with stealth mode to get you access to otherwise restricted urls.
Re: Cloudproxy – hide your scrapers IP behind the cloud
#28This is relatively easy to block, given that these companies publish their IP ranges. Similar to the Tor block, blocking "datacenter IP's" simply becomes the norm. This is why you have companies offering "resi's" or residential proxies to bypass such blocks for some years now.
I heard a while back, on HN 8 I think, that a lot of residential proxies are computers that have been infected with malware. How true is that?
A significant portion are grey market "ISPs" that purport to sell residential services but actually never do.
They sell these "residential" IPs to Amazon, other ecommerce retailers and shady people for an extreme price.
In the e-commerce world, scraping is necessary to stay in business. Amazon has armies of scrapers constantly monitoring their competitors, and in some cases automatically undercutting price updates.
Re: Cloudproxy – hide your scrapers IP behind the cloud
#29This is relatively easy to block, given that these companies publish their IP ranges. Similar to the Tor block, blocking "datacenter IP's" simply becomes the norm. This is why you have companies offering "resi's" or residential proxies to bypass such blocks for some years now.
Are there even ethical or “official” ways to source residential IPs? I thought most of them were from botnet infected IoT devices.
You can buy a bunch of mobile SIMs and use them.
Re: Cloudproxy – hide your scrapers IP behind the cloud
#30Earlier quoted context omitted.
It’d be easier to use Cloudflare WARP command line client for the same purpose. Apple uses Cloudflare for their private relay so there’s a chance that these IPs won’t be blocked.
If I remember correctly, Cloudflare WARP forwards your client IP in the header. So it probably won't be useful for scraping.
That's correct. More detail here: https://news.ycombinator.com/item?id=21070828