YQL is surprisingly quite brilliant:
Ask HN: What info do you web scrape for?
21–30 of 115 posts
Re: Ask HN: What info do you web scrape for?
#22I scrape Gumtree and eBay hourly using a python script for certain things I want under a certain price. The script sends me an email with the link in it and I get on top of it sharpish.
Managed to bag a lot of stuff over the last couple of years for not much money.
If someone bags this up as a service I'd pay for it.
Re: Ask HN: What info do you web scrape for?
#23I'm working on a startup that has web scraping at its core. The vision is a bit larger and includes fusing data from various sources in a probabilistic way (e.g. the same people, products, or companies found on different sides with ambiguous names and information. This is based on the research I've doen at uni). However, I found that there are no web crawling frameworks out there that allow for large-scale and contin…
How do you use a probabilistic approach to scraping data? Were you able to get a low number of false positives?
Sorry for the confusion. They are used for "merging" scraped data from various sources, not in the scraping process itself. For example, they help in figuring out if similar-sounding listings on related websites refer to the same "thing".
If interested, take a look at this (and related) papers: http://www.cs.ubc.ca/~murphyk/Papers/kv-kdd14.pdf
Re: Ask HN: What info do you web scrape for?
#24When I worked at MyEdu, I didn't actually sign on with the dev team originally — I worked on "the scraper team". We scraped college and university websites to get class schedule information: which classes were being taught, broken down by department and course number; by which professors; at which times on which days. If you're ever looking for an interesting challenge, I would encourage you to try getting this data.
Well-formed HTML is the exception rather than the rule and page navigation is often "interesting". Sometimes the school's system will use software from companies like Sungard or PeopleSoft, but there's customization within that... and of course, there's no incentive for the schools to aggregate this information in a common format (hence MyEdu's initiative), so there are plenty of homegrown systems. In short, there's no one-size-fits-all solution.
* NOTE: If you do attempt this, I insist that you teach throttling techniques from the very start. Some schools will IP block you if you hit them too hard; other schools have crummy infrastructure and will be crushed by your traffic. Scrape responsibly!
Re: Ask HN: What info do you web scrape for?
#25A lot of services with online billing refuse to send bills by e-mail, instead requiring users to log into their websites.
No doubt the companies would justify this by saying e-mail isn't secure enough. The side-effect that it'll stop many users bothering to look at their bill isn't why they do it at all, no sir.
I've been considering making a web scraper that goes to the phone company, electricity company, gas company, broadband company, electronic payslips, bank, stockbroker, AWS and so on; logs in with my credentials; downloads the PDF (or html) statements; and sends them by e-mail.
Of course, such a web scraper would need my online banking credentials, so I'm not in the market for a software-as-a-service offering.
Re: Ask HN: What info do you web scrape for?
#26I once wrote a scraper for a Yellow Pages site in Python. It pulled down the business category, name, telephone and email for every entry, and returned a nicely formatted spreadsheet. The hours I spent learning the ElementTree API and XPath expressions have paid for themselves several times over, now that I have a nicely segmented spreadsheet of business categories and email addresses, which I target via email marketing.
Re: Ask HN: What info do you web scrape for?
#27Reddit and twitter account for tendancy.
but twitter is so vast you may want to categorize account.
But reddit is a good source for a lot of info.
Re: Ask HN: What info do you web scrape for?
#28I've had three primary uses of web scraping. The hard part for me has never been speed. Getting the results structured is somewhere between easy and hideously complicated. 1. Reformatting and content archival (lag times of hours to days are no prob). As an example, I put together a site to archive comments of a ridiculously prolific commenter on a site I follow. I needed the content of his comments, as well as the tr…
How can I contact you?
Re: Ask HN: What info do you web scrape for?
#29A while ago, I had the idea of creating a travel site that catered to the group of people that enjoy traveling but aren't bound by time (i.e. I want to go to X, but I don't care when -- just show me the cheapest weekend for the next 3 months).
Anyway... it turns out that flight APIs are ridiculously non-existent. I ended up scraping two different airline sites, but since it was against their terms, I never took the site any further.
Re: Ask HN: What info do you web scrape for?
#30I scrape about 60 blogs and news sites that deal with a niche topic and examine all the hyperlinks. If more than one of them links to the same page, I assume that it's a page that's generating some buzz, so I send it out in an email. It's proved to be a generally reliable assumption.
Would love it if you posted a blog article sometime about the technical details.
There's really not much to it. Scrape each site/feed every X minutes, find all the hyperlinked URLs on the page, add them to a database table ... and if they're already in the table, send out an email with links to the "buzzed about" URL, as well as all of the sites/feeds that mention it. I keep the links in the table for about a month.