Diffbot offers a series of awesome automatic APIs for data extraction -- no setting up manual rules, just provide a URL you want to extract data from and they'll visually process and extract automatically. Also provide a crawler and bulk extractor, and a free trial -- www.diffbot.com
Show HN: CloudScrape – Cloud-based web scraping platform
51–60 of 109 posts
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#52Hit a certain threshold and you'll most likely get IP banned by the site you're scraping or heavily throttled unless the site does not care to minimize bot traffic (which can cost the site owner valuable bandwidth and server resources) You may also get sued... If the site owner wants its data to be available for automated extraction they'd provide an API and can price it to compensate for cost of serving all those bo…
If everyone had such a negative attitude as that we never would have had Google. Countless sites depend on web scraping. You can scrape and be a good netizen, the two are not mutually exclusive.
Yeah , lol , try to scrap Google search results and see how Google has such a positive attitude. not.
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#53Earlier quoted context omitted.
That's not a good argument. Do you ever leave your stuff lying around? I guess we can just take it then right? Just because you have access to something doesn't give you permission to access or access it in any manner possible.
That is not how HTTP works; your analogy is not correct. Nobody is taking anything. If you don't want someone to access your page, then don't respond to their request.
At a high enough frequency, scraping is indistinguishable from a DDoS attack. Do you believe DDoS attacks are OK? How do you draw the line?
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#54Probably a super dumb question, but isn't this fairly unethical? The "automatic IP rotation" feature isn't there for no reason.
I don't think it's unethical to pull down a copy of public information. If you pull too fast it might be considered rude (heavy load on the server). That's why some sites reflexively block all scrapers, hence the rotating IP feature. Hopefully this tool is rate-limited so it's not rude.
In terms of copyright, what matters is what you do with the scrape. If you scrape a public website for personal use, it's no different from just browsing it for personal use. If you try to republish the content for your own benefit, you'll run afoul of copyright law.
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#55Earlier quoted context omitted.
Why? People will just scrape your data anyway. What you put online is public . Treat the web the same as a public bulletin board. Everything you post will be readable by everyone. Trying to prevent this is pointless.
Well that sure is an arrogant reply, and not even close to true. In the US everything published (online or not) is copyrighted. Since you're clearly not a content creator, and have decided since you didn't work on it it has no value, but putting something online does not make content public domain.
Putting it online allows me to scrape it, but not to republish it.
I might just want to train a neural network on extracting data out of websites, and, for that usecase, wget -m every website I can find.
How I use the data you present to me, as long as I don’t republish it, is my decision. If you give me a license to read it, you also give me the license to copy it onto up to 7 different media at the same time, and to show it to up to 7 friends at the same time.
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#56Earlier quoted context omitted.
You have the right to disagree, but that's the way World Wide Web was built. Feel free to use alternative service(s) or stop publishing your stuff. Put it behind password or don't answer to my scrapers or browsers requests. Fair and simple. Web is for people from people, not solely for company(s) financial interests.
What's the WWW or the way it's built have to do with it? And the web is just technology, it's not "for" anyone or anything in particular. Intentions matter - on both sides. This is what most of the legal framework of the entire world is based on. You can disagree with that but again the ability to do something doesn't grant permission to do it. You're saying the solution to that is to remove the ability, but I don't…
>What's the WWW or the way it's built have to do with it?
What it has to do with it is that putting an HTTP server on the public Web signals the intention to serve up resources to anyone who sends an HTTP request. Any restrictions to this default must be implemented explicitly on top of the default.
Leaving my stuff lying around does not signal my intention for anyone to take it, unless I let it lying around next to the bins.
So yes, intentions matter. The question is how we learn about them. Sometimes the choice of technology implies particular intentions by default.
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#57I miss yahoo pipes
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#58This can become prohibitively expensive even for a small web scraping job that you're required to do everyday. Say scraping a news site for articles. The tooling and intuitivity is awesome but besides that at the heart doesn't it do what any other headless javascript enabled browser does? For example: https://phantomjscloud.com/site/pricing.html Is there a way to get a less feature rich version of this i.e. sans auto…
Thank your for the kind words :) While you're right that we do run a headless browser-ish thing what sets us apart from most of our competitors, other than the point-and-click approach is that we autodetect everything that's going on in the browser - which also means that in most cases you dont need to know what's going on. What this effectively means is that you'll often spend no time reverse-engineering and be able to scrape even wildly complex javascript-heavy sites in minutes instead of hours - and have them be a lot more stable than they would otherwise since there's no "Wait for 5 seconds" which will only be enough 95% of the time.
We see a lot of our clients manage to do what they need done - even daily scrapes - for as little as $29 / month since scraping a news site daily will often take up no more than a few minutes.
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#59Nice timing, as just this week I had my first exposure to the Cloudscrape platform. Credit where credit is due, I found the platform refreshingly feature rich even at lower tier (i.e. free) account levels compared to other offerings from competitors. Downsides were like so many other crawlers/spiders - too much in trying to meet the needs of all. Aiming for the 95% is easy, 95 to 99% though, not so much. Hence all th…
Thank you for the recommandations! Would love to know more about your exact use cases - we're aiming for the 100% :) If you'd reach out on support@cloudscrape.com - we'd be happy to investigate if we can't make CloudScrape work for you as well.
Re: Show HN: CloudScrape – Cloud-based web scraping platform
#60I miss yahoo pipes
We've got something in store that might make you happy then. Please stay tuned :)
https://twitter.com/cloudscrape