I just started getting into scaping (mostly been using import.io) mostly because it is a complement to what I really care about - data visualization. I've gotten a ton of interest for my side project and despite that I haven't opened the beta I'm still worried that it won't be as lucrative as creating some niche reporting services for various verticals (real estate, auto, etc). Essentially data that is very tabular and not hierarchical or qualitative. You can think of my work as pivot charts on crack. If someone already pre-compiled this data, I'd much rather pay for it that do it myself. My value add is the analysis/viz done on top of the data. If you want to chat, feel free to email me, contact info is in profile.
Ask HN: What info do you web scrape for?
51–60 of 115 posts
Re: Ask HN: What info do you web scrape for?
#52I've had three primary uses of web scraping. The hard part for me has never been speed. Getting the results structured is somewhere between easy and hideously complicated. 1. Reformatting and content archival (lag times of hours to days are no prob). As an example, I put together a site to archive comments of a ridiculously prolific commenter on a site I follow. I needed the content of his comments, as well as the tr…
There are services that cover at least part of what you mentioned. These effectively provide you a tool to visually build a scraper and then they automate the scraping in the background, creating an API or spreadsheet of the data.
Import.io is one example, and I think there's another more recent YC-backed one. However, I tried using import.io a little while back but without much joy.
Re: Ask HN: What info do you web scrape for?
#53I scrape Google to save search results.
Re: Ask HN: What info do you web scrape for?
#54Partial plug, but very related to the topic: if you're doing large scale analysis on the web and you don't want to have to actually run a large scale crawl, use the CommonCrawl dataset[1]! Common Crawl is a non profit organization that wants to allow anyone to use big web data.
I'm one of the team behind the crawl itself. Last month (July) we downloaded 4 billion web pages. Thanks to Amazon Public Datasets, all of that data is freely distributed via Amazon S3, under a very permissive license (i.e. good for academics, start-ups, businesses, and hobbyists). If your hardware lives on EC2, you can process the entire thing quickly for free. If you have your own cluster and many many terabytes of storage, you can download it too!
People have used the dataset to generate hyperlink graphs[2], web table content[2], microdata[2], n-gram and language model data (ala Google N-grams)[3], NLP research on word vectors[4], and so on, so there's a lot that can be done!
[1]: http://commoncrawl.org/ [2]: http://webdatacommons.org/ [3]: http://statmt.org/ngrams [4]: http://nlp.stanford.edu/projects/glove/
Re: Ask HN: What info do you web scrape for?
#55Earlier quoted context omitted.
What about legal implications? Do you get permission from the sites you crawl?
[deleted]
Thank you for answering. Though private and internal, you are still using the data for profit, is that correct? Does it mean that one can't sell the data, but can use it for analysis etc and still profit from it?
I saw a service recently that emails app store reviews/ratings for a fee. Not sure if they are scraping or getting the reviews some other way. The same idea can be extended to lots of things like Amazon reviews etc. Not sure of the legal stuff though.
Re: Ask HN: What info do you web scrape for?
#56Reddit and twitter account for tendancy. but twitter is so vast you may want to categorize account. But reddit is a good source for a lot of info.
What is tendancy ?
Re: Ask HN: What info do you web scrape for?
#57I scrape craigslist for side by side comparisons of stuff I want to buy from there. Eg: Cars, motorcycles, etc. Maybe even real estate would be a good target.
Re: Ask HN: What info do you web scrape for?
#58I love scrapping and even made a subreddit for the purpose where I showcased few of my public work. Any scraping lover can join in.
Re: Ask HN: What info do you web scrape for?
#59I've had three primary uses of web scraping. The hard part for me has never been speed. Getting the results structured is somewhere between easy and hideously complicated. 1. Reformatting and content archival (lag times of hours to days are no prob). As an example, I put together a site to archive comments of a ridiculously prolific commenter on a site I follow. I needed the content of his comments, as well as the tr…
There are services that cover at least part of what you mentioned. These effectively provide you a tool to visually build a scraper and then they automate the scraping in the background, creating an API or spreadsheet of the data. Import.io is one example, and I think there's another more recent YC-backed one. However, I tried using import.io a little while back but without much joy.
We have a non-tech intern and import.io looks like like a great tool to get him chewing up data. I'm playing with it now. Why didn't it work out for you? Beyond the wrapped browser interface being a little funky lol. (Edit: eugh, selecting data for import is really clunky.)
Ask HN: Anybody got a visual scraping service they like?
Re: Ask HN: What info do you web scrape for?
#60Partial plug, but very related to the topic: if you're doing large scale analysis on the web and you don't want to have to actually run a large scale crawl, use the CommonCrawl dataset[1]! Common Crawl is a non profit organization that wants to allow anyone to use big web data. I'm one of the team behind the crawl itself. Last month (July) we downloaded 4 billion web pages. Thanks to Amazon Public Datasets, all of th…
What would it take in terms of resources, etc. to get to the point where Common Crawl was doing web scale crawls on a regular basis?