Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: CloudScrape – Cloud-based web scraping platform

cloudscrape.com

Post reply on HN
Print view View original

101–109 of 109 posts

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#101

Post by kuschku »

troymc wrote: post #100
kuschku wrote: post #99

Earlier quoted context omitted.

If I republish, yes. If I just download a site – for local use, or similar cases – it’s not an issue. And if I provide a scraper where the user customly has to enter the URL, it’s also technically just a very advanced webproxy, and also okay.

That is incorrect. Even if you don't republish the data, the scraping alone can be a violation of the site's Terms of Service, and you can be sued (and lose). See the Wikipedia article I linked above. Think about it from the point of view of the site being scraped. Suddenly your webserver is being slammed with traffic that's way beyond what any human would cause. It's costing you bandwidth and compute resources. You…

And where I am, courts decided that no, ToS are never binding. ToS are effectively worthless.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#102

Post by manigandham »

fauigerzigerk wrote: post #93
manigandham wrote: post #91

Earlier quoted context omitted.

Putting a up a webserver that can be publically reached is not authorization to access it. I really can't say this in any other way - just because you can do something doesn't mean you are allowed to, whether it's online or offline. We already have an explicit signal called robots.txt which major search engines use. The problem is that there's no way to enforce this and there's just very little enforcement against ac…

" >Putting a up a webserver that can be publically reached is not authorization to access it. " This is legally incorrect. Without any further information or protection measures by the publisher it is legal to access content on a public web server. " >I really can't say this in any other way - just because you can do something doesn't mean you are allowed to " You are allowed to do everything that is not expressly fo…

1) You're not a lawyer. https://news.ycombinator.com/item?id=10339328

2) This directly contradicts your previous comment: "Leaving my stuff lying around does not signal my intention for anyone to take it, unless I let it lying around next to the bins."

3) There is already case law precedent regarding this exact type of publically accessible information not being authorized: http://www.net-security.org/secworld.php?id=14614

I agree that this is about ethics and any scraper that doesnt honor robots.txt and explicitly uses different IPs, user-agents, and other methods to mainly disguise itself as a machine service is unethical in this context.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#103

Post by novaleaf »

kennethologist wrote: post #21

This can become prohibitively expensive even for a small web scraping job that you're required to do everyday. Say scraping a news site for articles. The tooling and intuitivity is awesome but besides that at the heart doesn't it do what any other headless javascript enabled browser does? For example: https://phantomjscloud.com/site/pricing.html Is there a way to get a less feature rich version of this i.e. sans auto…

Thanks for the link to my service! (PhantomJsCloud)

FYI, I have a public preview of our new version, totally free. Check it out if you find this kind of service interesting! http://stage2.phantomjscloud.com

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#104

Post by chinathrow »

kuschku wrote: post #98
chinathrow wrote: post #97

Earlier quoted context omitted.

Well, copyright does prohibit redistributing the scraped content. And that is what all 3rd party content scraping services essentially do. That violates copyright law in quite a few countries. If you scrape with your own tools, own servers and use it only within your company, I would think it's a different legal matter. No, it's not "by law" public, or could you cite a few laws for that, both US and EU. Would appreci…

It is public because the HTTP standard defines it as public, and, if you claim to use this standard, its definitions apply to you (if you claim to support a standard, you have to accept that its definitions will be used in court against you) HTTP directly says you should use authorization to make stuff non-public. And third-party content scraping tools are NOT violating copyright law, as they do not redistribute it f…

You can have your content blocked from archive.org also by robots.txt - didn't check archive.is

Are you a lawyer? I am not - anyway, I can't really agree with that falls under fair use.

If I charge money to scrape someones site and then hand over the material in whatever form my customer uses - I am bound run into problems sooner or later.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#105

Post by fauigerzigerk »

manigandham wrote: post #102
fauigerzigerk wrote: post #93

Earlier quoted context omitted.

" >Putting a up a webserver that can be publically reached is not authorization to access it. " This is legally incorrect. Without any further information or protection measures by the publisher it is legal to access content on a public web server. " >I really can't say this in any other way - just because you can do something doesn't mean you are allowed to " You are allowed to do everything that is not expressly fo…

1) You're not a lawyer. https://news.ycombinator.com/item?id=10339328 2) This directly contradicts your previous comment: "Leaving my stuff lying around does not signal my intention for anyone to take it, unless I let it lying around next to the bins." 3) There is already case law precedent regarding this exact type of publically accessible information not being authorized: http://www.net-security.org/secworld.php?id…

>You're not a lawyer

And you are a lawyer?

>This directly contradicts your previous comment

Absolutely not. The default intention of putting an HTTP server online is not "letting stuff lying around", it is publishing stuff. And yes, the default can be overruled in various ways.

>There is already case law precedent regarding this exact type of publically accessible information not being authorized

You're grasping for straws here. In this specific case, it was completely obvious that this information was not supposed to be public. It was an embarrassing security failure that the defendant wanted to expose.

I think we agree on a lot. robots.txt should be honored and scraping in way or for a purpose that negatively impacts the website's viability or business model is unethical. But usually, such purposes are covered by copyright law anyway.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#106

Post by snowwrestler »

chinathrow wrote: post #88
snowwrestler wrote: post #54

Earlier quoted context omitted.

I don't think it's unethical to pull down a copy of public information. If you pull too fast it might be considered rude (heavy load on the server). That's why some sites reflexively block all scrapers, hence the rotating IP feature. Hopefully this tool is rate-limited so it's not rude. In terms of copyright, what matters is what you do with the scrape. If you scrape a public website for personal use, it's no differe…

If you disobey robots.txt it's unethical and rude.

I agree, but running a scraper does not necessarily mean disobeying robots.txt.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#108

Post by Richdow »

cookiecaper wrote: post #70
x5n1 wrote: post #67

Earlier quoted context omitted.

Keep your prices, your average developer is not a business person and advises everyone to race to the bottom.

Absolutely. People will always tell you they want everything for either practically or actually free. What they say they'll pay and what they'll actually pay are usually two very different things, assuming you provide something that can't be easily replaced.

Agreed! Brilliant tool!

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#109

Post by Richdow »

juanescobarcom wrote: post #51

Diffbot offers a series of awesome automatic APIs for data extraction -- no setting up manual rules, just provide a URL you want to extract data from and they'll visually process and extract automatically. Also provide a crawler and bulk extractor, and a free trial -- www.diffbot.com

Have you tried Cloudscrape? From the looks of it they can do the same, and then more?
Post reply on HN