Reddit and twitter account for tendancy. but twitter is so vast you may want to categorize account. But reddit is a good source for a lot of info.
Why not use PRAW? It's very mature, useful library using the Reddit API.
Ask HN: What info do you web scrape for?
31–40 of 115 posts
Re: Ask HN: What info do you web scrape for?
#32I've had three primary uses of web scraping. The hard part for me has never been speed. Getting the results structured is somewhere between easy and hideously complicated. 1. Reformatting and content archival (lag times of hours to days are no prob). As an example, I put together a site to archive comments of a ridiculously prolific commenter on a site I follow. I needed the content of his comments, as well as the tr…
What about legal implications? Do you get permission from the sites you crawl?
Re: Ask HN: What info do you web scrape for?
#33A while ago, I had the idea of creating a travel site that catered to the group of people that enjoy traveling but aren't bound by time (i.e. I want to go to X, but I don't care when -- just show me the cheapest weekend for the next 3 months). Anyway... it turns out that flight APIs are ridiculously non-existent. I ended up scraping two different airline sites, but since it was against their terms, I never took the s…
So, Skyscanner? (Pick'Whole month' / 'Whole year' for the date).
Re: Ask HN: What info do you web scrape for?
#34PlainSite (http://www.plainsite.org) uses about 20 different scrapers/parsers to download and standardize legal materials.
Re: Ask HN: What info do you web scrape for?
#35When I worked at MyEdu, I didn't actually sign on with the dev team originally — I worked on "the scraper team". We scraped college and university websites to get class schedule information: which classes were being taught, broken down by department and course number; by which professors; at which times on which days. If you're ever looking for an interesting challenge, I would encourage you to try getting this data.…
I've attempted to do this before for a side project (RateMyProfessor, but for textbooks!) and its incredibly hard to do accurately. One of the (many) issues that I ran into was that some schools still have all of their course data PDF format, in addition to the problems Cyranix listed above.
Much respect to any person or team that has to wade through this stuff.
Re: Ask HN: What info do you web scrape for?
#36I've had three primary uses of web scraping. The hard part for me has never been speed. Getting the results structured is somewhere between easy and hideously complicated. 1. Reformatting and content archival (lag times of hours to days are no prob). As an example, I put together a site to archive comments of a ridiculously prolific commenter on a site I follow. I needed the content of his comments, as well as the tr…
Thank you for the great feedback. I have a real estate background as well and keep wanting to find a project that would benefit that industry. It feels like a lot of what's out there is stuck in the past. I would love to help fix that. Sounds like I have a new project!
Re: Ask HN: What info do you web scrape for?
#37Scraping really is a quite complex process, and not everybody does it right.
Do you employ a (distributed?) crawler pool? What if a scraped page goes offline (404/410)? And, how do you handle network errors, and 403's / getting caught (and possibly blocked) - if at all? Do you conceal the scraping by employing a fake user agent? Do you (sometimes?) request permission for scraping to relevant webmasters?
These are the things that can make it or break it IMHO.
Re: Ask HN: What info do you web scrape for?
#38i'm building a database of games and scores. web scraping has been very helpful.
maybe instead of trying to change up the content, try to change up the method. ie. do a talk on running crawlers/scrapers to seed your database at an interval. (instead of just "scraping").
Re: Ask HN: What info do you web scrape for?
#39if you could publish a price list for items sold at major grocery chains, I am sure that many people could use it (bonus if it includes aisle numbers)
Not a bad idea, but I think the grocery chains are going to fight you on it.
Re: Ask HN: What info do you web scrape for?
#40We do so to determine new pages on websites within our industry. Often the new information here is not formally announced or done so only weakly. We regularly uncover valuable new info about company developments and changes.