- Auto-resolve CAPTCHA’s - Automatic IP rotation That's just wrong. Is there a website where we can blacklist IP addresses of such violators ?
I wonder if they respect noindex tags, and the like. If so, I would not see a problem.
Show HN: CloudScrape – Cloud-based web scraping platform
31–40 of 109 posts
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#32Hmm, another cool web scraper is import.io I use to think web scraping was limited to obscure proprietary companies like connotate so I'm glad to see more of these tools becoming available for everyone to utilize and hopefully create something cool.
Intercom.io is a very cool tool indeed. We use it to extract unique product links from product overview pages of webshops for adding them to Pricepin (our own tool).
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#33Probably a super dumb question, but isn't this fairly unethical? The "automatic IP rotation" feature isn't there for no reason.
As much as unethical as using adblock or disabling javascript. If you don't want your content to be scraped, don't put it online!
That's not a good argument. Do you ever leave your stuff lying around? I guess we can just take it then right?
Just because you have access to something doesn't give you permission to access or access it in any manner possible.
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#34Earlier quoted context omitted.
As much as unethical as using adblock or disabling javascript. If you don't want your content to be scraped, don't put it online!
That's not a good argument. Do you ever leave your stuff lying around? I guess we can just take it then right? Just because you have access to something doesn't give you permission to access or access it in any manner possible.
[deleted]
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#35Hit a certain threshold and you'll most likely get IP banned by the site you're scraping or heavily throttled unless the site does not care to minimize bot traffic (which can cost the site owner valuable bandwidth and server resources) You may also get sued... If the site owner wants its data to be available for automated extraction they'd provide an API and can price it to compensate for cost of serving all those bo…
If everyone had such a negative attitude as that we never would have had Google. Countless sites depend on web scraping. You can scrape and be a good netizen, the two are not mutually exclusive.
Google has a unique position in that any site that wants to be found has to let Google bot index its content. Google does not build a derived product from your site's content that ends up competing with your site.
If someone needs to build a search site for real estate etc why couldn't they just scrape the Google search result, filter it (white list domains), extract the actual link and present it? In that case you'l need a Google specific scraper that can be based on open source scraping libraries.
Update: Google actually will IP ban you if it thinks you're a bot trying to scrape for search results.
But they have an API:
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#36Earlier quoted context omitted.
As much as unethical as using adblock or disabling javascript. If you don't want your content to be scraped, don't put it online!
That's not a good argument. Do you ever leave your stuff lying around? I guess we can just take it then right? Just because you have access to something doesn't give you permission to access or access it in any manner possible.
That is not how HTTP works; your analogy is not correct.
Nobody is taking anything. If you don't want someone to access your page, then don't respond to their request.
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#37- Auto-resolve CAPTCHA’s - Automatic IP rotation That's just wrong. Is there a website where we can blacklist IP addresses of such violators ?
Google's business is to crawl all websites, but they IP block crawlers from their website. IP rotation is a solution to crawl such hypocrites.
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#38Earlier quoted context omitted.
Why? People will just scrape your data anyway. What you put online is public . Treat the web the same as a public bulletin board. Everything you post will be readable by everyone. Trying to prevent this is pointless.
It might be desirable to make data available to humans, but not bots, no?
You're making a distinction that, if you were to actually explain it in a way that makes sense, would reveal a much more relevant solution to this problem.
What's the difference between a machine scraper and a human scraper? Humans use machines to access websites. All your users are machines. And behind every machine is a human.
If you don't want someone to access all your data in a short time-frame, then don't give it all away in a short time-frame.
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#39I built a little Swift/iOS "search engine". It contains about 2200 urls. I only search page titles and tags, that I've manually added. What would be my best option for crawling the links and allowing the search to include the text of each page?
http://www.h4labs.com/dev/ios/swift.html
This is a weekend project so I don't want to spend a lot of money on it.
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#40Earlier quoted context omitted.
As much as unethical as using adblock or disabling javascript. If you don't want your content to be scraped, don't put it online!
That's not a good argument. Do you ever leave your stuff lying around? I guess we can just take it then right? Just because you have access to something doesn't give you permission to access or access it in any manner possible.
You have the right to disagree, but that's the way World Wide Web was built. Feel free to use alternative service(s) or stop publishing your stuff. Put it behind password or don't answer to my scrapers or browsers requests. Fair and simple.
Web is for people from people, not solely for company(s) financial interests.