Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudproxy – hide your scrapers IP behind the cloud

github.com

Post reply on HN
Print view View original

51–60 of 116 posts

Re: Cloudproxy – hide your scrapers IP behind the cloud

#51

Post by beebeepka »

Dunedan wrote: post #49
superasn wrote: post #27

Even though I only do it for hobby projects, crawling pages is becoming increasingly difficult unless you are a big player like Google or Microsoft with a whitelisted IP range. I've had some success in scraping lately with a similar project called FlareSolverr(1). It's purpose it to get you access to sites which won't let you crawl unless you are using a real browser (e.g amazon, instagram). It doesn't hide your IP b…

For one pet project I had to crawl a rather popular site. While that worked in general I would frequently get internal server errors. Turns out that this was the response of their CDN when it detected bot-like behavior. That left me wondering how they prevent search engine crawlers from being detected as bots and getting throttled this way as well. Turns out they just check the user agent for that. As soon as I put "…

Changing user agent to Google it or whatever also lets you past quite a few paywalls too. Or at least it used to. These days I don't even bother

Re: Cloudproxy – hide your scrapers IP behind the cloud

#52

Post by aeyes »

gibson1 wrote: post #5

i tend to use tor network as a proxy which enables me to request a new ip whenever i need

Tor has a couple of hundred exit nodes and they are all on public lists, the Tor network is easy to identify and block.

If you are in a position were you need a rotating list of proxy servers in the cloud to scrape, Tor is probably way too slow.

Re: Cloudproxy – hide your scrapers IP behind the cloud

#54

Post by abc03 »

samjmck wrote: post #17
tushar-r wrote: post #10

This is relatively easy to block, given that these companies publish their IP ranges. Similar to the Tor block, blocking "datacenter IP's" simply becomes the norm. This is why you have companies offering "resi's" or residential proxies to bypass such blocks for some years now.

Are there even ethical or “official” ways to source residential IPs? I thought most of them were from botnet infected IoT devices.

Packetstream.io buys bandwidth from users and is cheap. I‘m a customer.

Re: Cloudproxy – hide your scrapers IP behind the cloud

#55

Post by wu_187 »

holstvoogd wrote: post #18
TheFreim wrote: post #15

Is it ethical to scrape things, trying to bypass prevention methods, against the expressed wishes of a site?

As someone who has to deal with a lot of bots, bot networks and other weird scraper apps people use: The biggest issue is that most of these tools are not behaved very well. This tool is clearly designed to circumvent protections, rate limits mostly, against scraping that might be essential to keep things running. They follow links that are explicitly marked as do not follow, they do not even try to limit their rate,…

Exactly. We'd get customers thats sites would drown in bot traffic because they pulled the same shit, changing user agents, different ips, etc. I had to build custom mod security rules to block the patterns these boys would pull. What's funny is that the bots would have a site where you could control the crawl rates but it's just a placebo. They would crawl even if you requested them to stop.

The issue with bots hosted on AWS or any cloud for that matter is that as a web host you can't just block the IPs because legitimate traffic comes from them in the form of CMS plugins, backups, etc.

Re: Cloudproxy – hide your scrapers IP behind the cloud

#56

Post by hakube »

blantonl wrote: post #34

Unpopular opinion around here, but scraping Web pages is often unethical, and the scourge of many Web developers and infrastructure managers. For Christ sakes people, develop your own products! If you have to develop and use a cloud based tool to "get around" businesses blocking your scraping app, maybe reevaluate your business and life choices.

Most use cases for web scraping is usually related to data analysis, creating machine learning dataset, etc. Google and other search engine services are scraping the web too.

Re: Cloudproxy – hide your scrapers IP behind the cloud

#57

Post by meltedcapacitor »

blantonl wrote: post #34

Unpopular opinion around here, but scraping Web pages is often unethical, and the scourge of many Web developers and infrastructure managers. For Christ sakes people, develop your own products! If you have to develop and use a cloud based tool to "get around" businesses blocking your scraping app, maybe reevaluate your business and life choices.

Users are also victims: captcha walls are basically an anti-scrapper countermeasure, which everybody pays for in reduced web usability.

"Proof of work" style countermeasures (where data has to be decoded browser-side in expensive ways that are bearable for a regular user but onerous for a mass scraper) is another externality everybody pays for (the total amount of CPU wasted by regular users is at the system level a total waste).

Login walls...

Re: Cloudproxy – hide your scrapers IP behind the cloud

#59

Post by meltedcapacitor »

nlh wrote: post #44
blantonl wrote: post #39

Earlier quoted context omitted.

no business is under any obligation to provide you access to their data and services for you to build a business on top of. If you feel that the owner of data or services maintains an illegal monopoly over that data, work to lobby government to resolve that issue. It's not your individual place to unilaterally decide that a business owes you anything other than what is described in their terms and conditions of use.…

I used to have a similar attitude - most typically applied to rules & laws, etc. “If you don’t like the rule, go run for office and get the law changed.” But then I realized that’s just a cop out. A lot of structures and systems are in place specifically to make it difficult for people to change things and specifically to maintain monopolies. Nobody’s going to lobby government to resolve this issue. That’s the point.…

In the analog age, imagine someone made a card index of some interesting data you can come in a shop to consult, for a fee.

Would breaking into their premises to steal or copy the cards be on the same side of the line as the digital variant?

Re: Cloudproxy – hide your scrapers IP behind the cloud

#60

Post by meltedcapacitor »

pacifika wrote: post #2

Maybe it’s a better idea to start or subscribe to an ethical scraping initiative with some user agent header that references some kind of policy and verifiable membership id

That's the existing default: if a service doesn't mind scraping, they need not take any countermeasure (other than simple rate limiting with a clear status code). Any site that does go beyond that is telling data thieves that they're not welcome.
Post reply on HN