Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudproxy – hide your scrapers IP behind the cloud

github.com

Post reply on HN
Print view View original

41–50 of 116 posts

Re: Cloudproxy – hide your scrapers IP behind the cloud

#41

Post by iakh »

blantonl wrote: post #39
arsome wrote: post #38

Earlier quoted context omitted.

Sure, let me just develop my own monopoly on all real estate information.

no business is under any obligation to provide you access to their data and services for you to build a business on top of. If you feel that the owner of data or services maintains an illegal monopoly over that data, work to lobby government to resolve that issue. It's not your individual place to unilaterally decide that a business owes you anything other than what is described in their terms and conditions of use.…

Or if you don't want info public, don't make it public then complain people are using it. If you want to stop scrapers, offer an api. Nobody scrapes by choice if there's a supported alternative to getting the same data.

Re: Cloudproxy – hide your scrapers IP behind the cloud

#43

Post by arsome »

blantonl wrote: post #39
arsome wrote: post #38

Earlier quoted context omitted.

Sure, let me just develop my own monopoly on all real estate information.

no business is under any obligation to provide you access to their data and services for you to build a business on top of. If you feel that the owner of data or services maintains an illegal monopoly over that data, work to lobby government to resolve that issue. It's not your individual place to unilaterally decide that a business owes you anything other than what is described in their terms and conditions of use.…

> no business is under any obligation to provide you access to their data and services for you to build a business on top of.

I mean, they're the ones who gave it to me, I'm just using it. Not my problem if they don't like it. I never saw any terms and conditions when my browser loaded the page, no reason I'm going to go looking for them before I scrape it. I have just as much right to access it as my web browser does or my phone does.

I'm not saying they owe me anything, they're offering it to me, I'm just taking it.

Re: Cloudproxy – hide your scrapers IP behind the cloud

#44

Post by nlh »

blantonl wrote: post #39
arsome wrote: post #38

Earlier quoted context omitted.

Sure, let me just develop my own monopoly on all real estate information.

no business is under any obligation to provide you access to their data and services for you to build a business on top of. If you feel that the owner of data or services maintains an illegal monopoly over that data, work to lobby government to resolve that issue. It's not your individual place to unilaterally decide that a business owes you anything other than what is described in their terms and conditions of use.…

I used to have a similar attitude - most typically applied to rules & laws, etc. “If you don’t like the rule, go run for office and get the law changed.”

But then I realized that’s just a cop out. A lot of structures and systems are in place specifically to make it difficult for people to change things and specifically to maintain monopolies.

Nobody’s going to lobby government to resolve this issue. That’s the point. Real estate companies are super happy to keep the status quo.

Sometimes you have to break some rules to innovate. There is a line of course (where you draw it depends on your own ethical code). But I certainly wouldn’t put “scraping real estate data” behind the line.

Re: Cloudproxy – hide your scrapers IP behind the cloud

#45

Post by temp8964 »

superasn wrote: post #27

Even though I only do it for hobby projects, crawling pages is becoming increasingly difficult unless you are a big player like Google or Microsoft with a whitelisted IP range. I've had some success in scraping lately with a similar project called FlareSolverr(1). It's purpose it to get you access to sites which won't let you crawl unless you are using a real browser (e.g amazon, instagram). It doesn't hide your IP b…

Doesn't Amazon have an API? Why everyone wants to scrape Amazon? Honest question.

Re: Cloudproxy – hide your scrapers IP behind the cloud

#48

Post by sschueller »

TheFreim wrote: post #15

Is it ethical to scrape things, trying to bypass prevention methods, against the expressed wishes of a site?

It's public not behind a paywall or login. Just don't hammer the network, that is what most are concerned about as it costs money and time.

Re: Cloudproxy – hide your scrapers IP behind the cloud

#49

Post by Dunedan »

superasn wrote: post #27

Even though I only do it for hobby projects, crawling pages is becoming increasingly difficult unless you are a big player like Google or Microsoft with a whitelisted IP range. I've had some success in scraping lately with a similar project called FlareSolverr(1). It's purpose it to get you access to sites which won't let you crawl unless you are using a real browser (e.g amazon, instagram). It doesn't hide your IP b…

For one pet project I had to crawl a rather popular site. While that worked in general I would frequently get internal server errors. Turns out that this was the response of their CDN when it detected bot-like behavior. That left me wondering how they prevent search engine crawlers from being detected as bots and getting throttled this way as well. Turns out they just check the user agent for that. As soon as I put "Googlebot" in my user agent the frequent errors vanished. So sometimes it's not about using the right IP addresses, but just the right keywords in your user agent. ;-)

Re: Cloudproxy – hide your scrapers IP behind the cloud

#50

Post by trutannus »

popotamonga wrote: post #31
Havoc wrote: post #19

Not following how this is helpful? From what I've seen the cloud IPs are the first to get blocked by anti-scraping tech. e.g. Built an amazon scraping thing at home. Worked fine. Deployed to cloud. Literally first request Amazon goes nope you're a bot

I have my crawlers running on a 24/7 notebook at my house because of that. No blocks whatever rate i use. Deployed to digital ocean runs for 5 minutes then blocked forever.

Essentially. All you're doing here is making it easy for the target sites to mass block a bunch of scrapers. Running it off your device is generally better.
Post reply on HN