101–110 of 115 posts
People who put in a few hours of work to take advantage of other people's hard work piss me off. :/
I'm looking to buy a house, and not all local estate agents post to Rightmove (or some post with a 24-hour delay). Trying to submit the search form on the agent's own hideous website, parse the results and get a standard structure between them all is hideous - I gave up in the end.
Once I have the data the challenge is then analysing it (geolocation, how long are the commute times, distance to amenities etc) which is its own separate challenge
I also did some e-commerce information scraping.
One of the most interesting one was for a data selling company. They asked me to collect data of geo information, disaster, finance, tweets etc. We used to apply ML and statistics to give forecast with historic data.
Earlier quoted context omitted.
@allegory could you share the the script with us ?
Would love to but not at the minute because it has hard coded credentials for eBay API in it. It's on my list as a TODO to tidy it up. Will stuff on github and post a Show HN on it soon :) I've got one that monitors amazon prices for sudden lows as well.
I scrape about 60 blogs and news sites that deal with a niche topic and examine all the hyperlinks. If more than one of them links to the same page, I assume that it's a page that's generating some buzz, so I send it out in an email. It's proved to be a generally reliable assumption.
I just started getting into scaping (mostly been using import.io) mostly because it is a complement to what I really care about - data visualization. I've gotten a ton of interest for my side project and despite that I haven't opened the beta I'm still worried that it won't be as lucrative as creating some niche reporting services for various verticals (real estate, auto, etc). Essentially data that is very tabular a…
Machete.io is awesome!
I am looking for data having list of all universities and associated colleges . I didn't found anything related to it anywhere
I've had three primary uses of web scraping. The hard part for me has never been speed. Getting the results structured is somewhere between easy and hideously complicated. 1. Reformatting and content archival (lag times of hours to days are no prob). As an example, I put together a site to archive comments of a ridiculously prolific commenter on a site I follow. I needed the content of his comments, as well as the tr…
Can you mention some of those domains? I'm interested. I had worked on one such project earlier, for a financial startup.
Been scraping a lot lately but mostly:
- government website for license holders
- creating lists of businesses for different segments (market research/analysis)
- using those lists to scrape individual sites and make analysis (how many use facebook/youtube/etc)
I am looking for data having list of all universities and associated colleges . I didn't found anything related to it anywhere
So like the dmoz.org or higher education?
But information about all colleges associated with university is generally available only on specific site of university, and many times those are in pdf.
Having said that, my above requirement(college associated with university) itself is secondary. I found the list of universities at many places , but not the list of colleges.
So for now just getting a list of all college with location is enough for me.