Looks good. I've used WebScraper.io (running in Chrome on AWS EC2 instances) before for many projects. It's quite poerwful and free. http://webscraper.io
Show HN: CloudScrape – Cloud-based web scraping platform
81–90 of 109 posts
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#82Earlier quoted context omitted.
That is not how HTTP works; your analogy is not correct. Nobody is taking anything. If you don't want someone to access your page, then don't respond to their request.
At a high enough frequency, scraping is indistinguishable from a DDoS attack. Do you believe DDoS attacks are OK? How do you draw the line?
DDoS attacks are malicious events that disrupt service. In almost 100% of cases, scrapers don't want to disrupt service, because they need the data they're scraping. They want to be able to continue to get it, so they won't do things that may harm their ability to do that (including presenting honest IPs and user agents).
Services like this one actually make scraper-related unavailability, which IMO is already greatly exaggerated, less likely, since there will be fewer amateurs trying to write their own bots and accidentally breaking things.
To the extent that a scraper harms the other business, the scraping company can be held civilly liable on several accounts without specifically bringing scraping as a practice into the picture. All that matters is that they damaged the target site's ability to operate, not that they were saving [portions of] the pages (that'd be a separate copyright claim, unrelated to the disruption of service).
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#83Are there open source alternatives to services like this and Diffbot? Boilerpipe and similar libraries are ok but targeted at article extraction.
Portia from Scrapinghub is 100% open source: http://scrapinghub.com/portia/
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#84I'm wondering about use cases for this (?) Also, do they respect robots.txt? And how are they going to avoid being blocked by websites that don't want to be scraped? (I guess it would be easy to determine the IP addresses of their scrape robots).
One of their features is "Automatic IP Address Rotation" so it may not be easy....
Use their free tier to scrape your own honeypot site, log the traffic and use for blocking.
Wouldn't be hard to file abuse@ reports to their hosting provider as well. Just pipe their scrapper IPs into WHOIS.
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#85Earlier quoted context omitted.
Well that sure is an arrogant reply, and not even close to true. In the US everything published (online or not) is copyrighted. Since you're clearly not a content creator, and have decided since you didn't work on it it has no value, but putting something online does not make content public domain.
You're confusing redistribution with scraping. Scraping publicly-accessible content is legal. Redistributing it is not (at least not in the US). Google scraps webpages as a core competency.
Google also adheres to the robots.txt standard. Most of the scrapers I block don't.
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#86I need to add their IP range to my blocklist.
Why? People will just scrape your data anyway. What you put online is public . Treat the web the same as a public bulletin board. Everything you post will be readable by everyone. Trying to prevent this is pointless.
Why?
My site, my robots.txt - if the scrapers do not obey, then I'll block them.
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#87I need to add their IP range to my blocklist.
If these guys are serious about their business they are using proxies to obfuscate who they are. There are many services now that are claiming to have a ton of IP addresses Luminati, Shadio.io and Nohodo are just a few examples.
If these guys are serious about their business, then they should behave like a good netizen and obey robots.txt directives.
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#88Probably a super dumb question, but isn't this fairly unethical? The "automatic IP rotation" feature isn't there for no reason.
I don't think it's unethical to pull down a copy of public information. If you pull too fast it might be considered rude (heavy load on the server). That's why some sites reflexively block all scrapers, hence the rotating IP feature. Hopefully this tool is rate-limited so it's not rude. In terms of copyright, what matters is what you do with the scrape. If you scrape a public website for personal use, it's no differe…
If you disobey robots.txt it's unethical and rude.
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#89It's great the see discussions going on here - would like to tie a few comments to the questions of ethical aspects of web scraping: As some has pointed out scraping is not exactly a new thing and a lot of the biggest sites out there are built on the basis of web scraping or crawling. We provide a tool and expect you use that tool while abiding the law - and if not we will of course shut your account down immediately…
If you are a good netizen, could you plese provide your user agent so I can block your bot on all sites I operate?
Thank you.
EDIT: found that in your FAQ:
"Since disclosing IP’s and user agents would allow anyone to identify all traffic coming from our system – we naturally never do."
That is the opposite of being a good netizen and I hope I'll be able to sue you once I find out your services are helping to scrape my content.
2nd EDIT: Found out that you reside in Denmark and therefore in the EU, that makes it way easier then.
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#90Earlier quoted context omitted.
As much as unethical as using adblock or disabling javascript. If you don't want your content to be scraped, don't put it online!
That's not a good argument. Do you ever leave your stuff lying around? I guess we can just take it then right? Just because you have access to something doesn't give you permission to access or access it in any manner possible.
[deleted]