if you could publish a price list for items sold at major grocery chains, I am sure that many people could use it (bonus if it includes aisle numbers)
Similar to http://www.aislefinder.com and http://www.supermarketapi.com
Ask HN: What info do you web scrape for?
81–90 of 115 posts
Re: Ask HN: What info do you web scrape for?
#82There is a closet industry for scraping any sort of data that can move markets. Fed, crop, weather, employment,etc. Anything that is released at a certain time on a fixed calendar, you can bet that multiple parties are trying to scrape it as fast as possible. If you can scrape this data( the easy part), put it in a structured format( somewhat hard) and deliver it in under a few seconds(this is where you get paid) the…
My understanding is that you need to deliver the data with a latency measured in the range of milliseconds, and even then that might not be fast enough due to direct access. Here are a couple articles in the WSJ --
"Speed Traders Get an Edge" - Feb 6, 2014 - http://online.wsj.com/news/articles/SB1000142405270230445090...
"Firm Stops Giving High-Speed Traders Direct Access to Releases" - Feb 20, 2014 - http://online.wsj.com/news/articles/SB1000142405270230377550...
Re: Ask HN: What info do you web scrape for?
#83I have a side-project which scrapes play-by-play data from NBA games to gain more insights into these games.
Here is an example of the (un-finished) side-project: http://recappd.com/games/2014/02/07
I'm far from the only person scraping this data. Look at sites liked http://vorped.com and http://nbawowy.com for even better examples.
Re: Ask HN: What info do you web scrape for?
#84I've had three primary uses of web scraping. The hard part for me has never been speed. Getting the results structured is somewhere between easy and hideously complicated. 1. Reformatting and content archival (lag times of hours to days are no prob). As an example, I put together a site to archive comments of a ridiculously prolific commenter on a site I follow. I needed the content of his comments, as well as the tr…
There are services that cover at least part of what you mentioned. These effectively provide you a tool to visually build a scraper and then they automate the scraping in the background, creating an API or spreadsheet of the data. Import.io is one example, and I think there's another more recent YC-backed one. However, I tried using import.io a little while back but without much joy.
I think import.io is buggy to say the least, having used it in the past to scrape some websites, it was pain to work with. Kimonolabs is still very lacking in terms of ability to handle different websites, it is very much limited to a certain portion of the web, it looks like they are more about creating APIs...APIs that people are supposed to find interesting and valuable but like the topic of this question, it seems like it's only valuable to someone who has a direct need for that dataset, by itsef would serve no interest to say.
Having private access to Scrape.it, I can say that it focuses strictly on making a great tool and the ability to scrape websites without costing a fortune. I know the founders and they are extremely dedicated to making a tool that can pretty much handle anything you throw at it like AJAX, Single page apps, crawling selected links and all sub links after the page. They've just begun adding login and form support so should be able to play with those as well very soon. It only supports csv output at the moment but hopefully they can make available like API output.
Re: Ask HN: What info do you web scrape for?
#85I wrote a daily scraper that scrapes about 15 sites plus (cheerio) formats it and sends it to me in a responsive email from the node server that does all. Runs from a cron daily. The point is efficiency I get all my updates and data for the day in a few minutes versus perusing for 2 hours. Its like a shot of espresso versus a pot of Joe.
What else I scrape for I can't share as its lucrative.
Re: Ask HN: What info do you web scrape for?
#86I'm working on a startup that has web scraping at its core. The vision is a bit larger and includes fusing data from various sources in a probabilistic way (e.g. the same people, products, or companies found on different sides with ambiguous names and information. This is based on the research I've doen at uni). However, I found that there are no web crawling frameworks out there that allow for large-scale and contin…
>I found that there are no web crawling frameworks out there that allow for large-scale and continuous crawling of changing data. Are you distinguishing between "I found that there are no" and "I didn't find any so far"? Which ones that came close have you rejected, and why?
I can't say for sure that there are none, but I believe that I've done quite a bit of research. If there really was an excellent web crawling framework it should have bubbled up to the top.
I don't remember the names of all projects that I've looked at, but the main ones were Nutch, Hetrix, scrapy and crawler4j. I've come across several companies/startups that have built their crawlers in-house for the same reasons (e.g. http://blog.semantics3.com/how-we-built-our-almost-distribut...).
Re: Ask HN: What info do you web scrape for?
#87Earlier quoted context omitted.
Regarding 2) Why wouldn't it work for PDF's? If you're able to get the file itself, you should be able to OCR it... Is there anything obvious that I am missing in regards to PDFs?
It's not the text that's the issue, it's the structure. PDFs have nowhere near as much structure as markup. You end up having to do this for dozens of layouts and it gets hurty really fast: http://schoolofdata.org/2013/06/18/get-started-with-scraping...
There are computer vision libraries that automatically extract tables from PDFs. For example, http://ieg.ifs.tuwien.ac.at/projects/pdf2table/.
You may want to give that a try if you haven't looked at it before.
Re: Ask HN: What info do you web scrape for?
#88I once wrote a scraper for a Yellow Pages site in Python. It pulled down the business category, name, telephone and email for every entry, and returned a nicely formatted spreadsheet. The hours I spent learning the ElementTree API and XPath expressions have paid for themselves several times over, now that I have a nicely segmented spreadsheet of business categories and email addresses, which I target via email market…
As someone responsible for search on a yellow pages company, I can confirm that most YP websites have little to no protection against this. Company information is usually public anyway. We just make it very easy for you to get it :)
Re: Ask HN: What info do you web scrape for?
#89I'm working on a startup that has web scraping at its core. The vision is a bit larger and includes fusing data from various sources in a probabilistic way (e.g. the same people, products, or companies found on different sides with ambiguous names and information. This is based on the research I've doen at uni). However, I found that there are no web crawling frameworks out there that allow for large-scale and contin…
Would love to hear more about your use case Denny--sending you a PM.
As for web-scale crawling of particular verticals such as products and news, you might want to try: http://www.diffbot.com/products/automatic/
We're planning on releasing support for jobs, companies, and people later.
(disclosure: I work there)
Re: Ask HN: What info do you web scrape for?
#90I do a lot of scraping for my day job. We have a business intelligence team that will build us reports that we need from the data that we have. However I find that this process is so incredibly slow and sometimes we only need to compile the data for a one-off project. I used to use vb.net for this as that's what I started learning programming with. Now I use python/requests/bs4 for all my scraping scripts. I've start…
At Diffbot, we have an automated discussion thread parser, currently in beta testing, that might be exactly what you need. Send me a note at mike@diffbot.com and I'd be glad to hook you up.
(disclosure: I work there)